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FOREWORD 


The  proceedings  and  papers  that  comprise  this  report  were  prepared  in 
support  of  Navy  Decision  Coordinating  Paper,  Performance  Aids  Test  and 
Evaluation  (NDCP-Z0828-PN)  under  the  sponsorship  of  the  Director  of  Educa- 
tion and  Training.  The  papers  were  presented  by  specialists  in  job  per- 
formance aid  (JPA)  technology  at  the  Invitational  Conference  on  the  Status 
of  JPA  Technology  held  February  23-25,  1977  by  the  Navy  Personnel  Research 
and  Development  Center  (NAVPERSRANDCEN) . The  findings  of  the  conference 
contributed  significantly  to  the  planning  and  conduct  of  the  Advanced 
Development  project. 


J.  J.  C LARKIN 
Commanding  Officer 
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SUMMARY  AND  RECOMMENDATIONS 


Summary 


An  invitational  conference  was  held  February  23-25,  1977  at  the  Navy 
Personnel  Research  and  Development  Center  (NAVPERSRANDCEN)  to  assess  the 
status  of  the  state-of-the-art  in  job  performance  aid  (JPA)  technology. 

The  objectives  of  the  conference  were:  (1)  to  define  the  current  utate- 

of-the-art  in  JPA  technology  as  related  to  personnel/training  implications; 

C2)  to  obtain  a better  conceptualization  of  problems  involved  in  acceptance 
and  utilization  of  JPA  throughout  the  personnel,  training,  and  maintenance 
communities;  and  (3)  to  recommend  methodologies  for  integrating  JPA  technology 
with  the  Navy  personnel  systems  for  the  1980s.  The  objective  of  this  report 
is  to  publish  the  six  papers  presented  at  the  meeting,  plus  an  introduction 
by  Dr.  R.  E.  Blanchard  of  NAVPERSRANDCEN  and  a final  paper  assimilating  find- 
ings and  conclusions  drawn  from  the  proceedings. 

In  the  Introduction,  Dr.  Blanchard  describes  the  current  need  for  approaches 
to  controlling  or  reducing  burgeoning  personnel  costs  in  the  military.  He  notes 
that,  although  (1)  data  have  been  collected  on  JPAs  for  over  20  years  to  dem- 
onstrate resultant  performance  increments  and  (2)  payoffs  have  been  forecasted 
on  the  order  of  $1  to  $2  billion  annually  by  each  service  in  personnel,  train- 
ing, and  maintenance  through  the  use  of  JPAs,  they  have  not  been  implemented 
in  other  but  piecemeal  fashion  by  any  of  the  military  services. 

The  first  paper,  presented  by  Dr.  John  Collins  of  the  Essex  Corporation, 
which  was  entitled  "Some  Perspectives  On  the  Job  Performance  Aids  Technology 
Base,"  outlined  a generally  broad  concept  of  JPA  for  consideration.  His 
description  of  major  conceptual  and  methodological  developments  in  such  areas 
as  motivation  theory,  job  structure,  and  learning  studies  stimulated  discussion 
well  beyond  the  textual,  highly  proceduralized  type  of  job  aid. 

The  next  paper,  delivered  by  Mr.  Theodore  Post  of  BioTechnology,  Inc., 
covered  a method  for  matching  systems  to  presentation  techniques  being  considered 
by  the  Navy  Technical  Information  Presentations  Program  (NTIPP).  The  approach, 
entitled  "Selection  of  Formats  and  Media  for  Presenting  Maintenance  Information," 
appeared  to  be  a good  start  toward  that  part  of  a program  manager's  guide, 
which  would  give  him  tradeoff  criteria  to  use  in  selecting  the  best  JPA 
technique  for  his  particular  system.  It  also  provides  a way  of  greatly 
simplifying  the  number  of  JPA  techniques  to  consider  by  using  only  two  basic 
categories,  classified  as  either  the  "directive"  or  the  "deductive"  type 
of  job  aid. 

The  paper  presented  by  Mr.  Frank  Johnson  of  the  REM  Company,  which  was 
entitled  "Problems  in  Procuring,  Producing,  and  Evaluating  JPAs,"  clearly 
illustrated  the  need  for  an  integrated  approach  to  the  acquisition  of  training 
materials  and  technical  data.  Also,  in  showing  the  difficulties  of  ensuring 
error-free  technical  information,  his  paper  suggests  that  JPA  system  designers 
should  not  attempt  to  make  the  maintenance  technician  totally  dependent  on 
the  JPA,  particularly  if  the  tasks  cover  complex  troubleshooting  of  equipment. 


"User  Problems  In  JPA  Utilization,"  the  paper  presented  by  Mr.  Reid  Joyce 
of  Applied  Science  Associates,  provided  an  excellent  status  report  on  the 
general  nonacceptance  of  JPAs  even  though  almost  every  experimental  evaluation 
has  shown  that  technicians,  both  experienced  and  inexperienced,  perform  with 
fewer  errors  on  the  job  when  using  a JPA.  His  report  was  particularly 
enlightening  because  it  revealed  that  almost  all  of  the  past  nonacceptance  of 
JPAs  can  be  attributed  to  people  in  the  system  who  are  not  direct  users  of  the 
aid  itself.  These  include  the  technician's  peers  and  supervisors,  nonmaintenance 
users  of  the  documentation  system,  and  all  those  connected  with  hardware  and 
logistics  problems  at  the  program  manager  level. 

Dr.  Edgar  Shriver  of  Klnton,  Incorporated,  in  his  paper  "New  Directions 
for  Information  Transfer  Research  in  Maintenance  Jobs,"  suggests  five  funda- 
mental elements  to  be  used  in  assessing  the  quality  of  any  JPA.  Four  of  the 
elements  are  types  of  content  analysis  described  as  "equipment  analysis," 
"functional  analysis,"  "task  analysis,"  and  "behavioral  task  analysis."  The 
fifth  fundamental  element,  "intelligibility,"  covers  all  format  considerations 
for  text  and  graphics.  His  paper  also  described  the  approach  the  Army  has 
initiated  toward  integrating  their  training  and  technical  data  systems. 

Mr.  John  Klesch,  of  the  Air  Force  Human  Resources  Laboratory,  presented  a 
paper  entitled,  "Implementation  of  the  JPA  Job-Oriented  Training  Approach  to 
Maintenance:  The  Impact  on  Personnel  Systems."  The  discussion  following  his 

presentation  addressed  how  JPA  experiments  in  the  past  have  adversely  affected 
individual  careers  in  the  Air  Force.  It  is  extremely  important,  therefore, 
that  JPAs,  when  introduced  into  existing  logistics  support  systems,  even  on  a 
test  basis,  make  necessary  corresponding  systems  changes  as  well. 

The  final  paper,  that  of  Dr.  Harold  Booher  of  NAVPERSRANDCEN,  concerned 
"Status  of  JPA  Technology:  Analysis  and  Conclusions."  Dr.  Booher  concluded 

that,  although  there  are  only  a limited  number  of  JPA  techniques  ready  for 
implementation  into  Navy  Weapon  Systems,  these  are  quite  powerful  and  well 
documented.  Thus,  if  they  are  introduced  with  corresponding  changes  in  person- 
nel and  training  systems,  they  can  be  expected  to  produce  the  payoffs  anticipated. 
There  are  still  a large  number  of  considerations  in  personnel  and  training,  how- 
ever, which  vlll  have  to  be  addressed  and  solved  within  a totally  new  personnel 
system  concept  before  the  full  potential  of  JPA  technology  can  be  realized. 

Recommendat ions 

In  Dr.  Booher' s paper,  the  following  recommendations  or  guidelines  were 
drawn  as  a result  of  conference  proceedings. 

JPA  Technology  Applications 

1.  Select  JPA  test  cases  for  maintenance  technical  information  presen- 
tation using  both  troubleshooting  and  nontroubleshooting  tasks. 

2.  Conduct  longitudinal  demonstration  (1-5  years)  of  each  test  case  to 
determine  extent  of  system  life  cycle  cost  payoffs  and  positive  influence  on 
personnel  careers. 


vlll 


3.  For  nontroubleshooting  tasks,  utilize  the  fully  proceduralized 
job  performance  aid  for  format/content  technique  with  paper  as  the  informs 
tion  transfer  medium. 


4.  For  troubleshooting  tasks,  utilize  a combination  of  directive 
and  deductive  JPA  formats/content  techniques  with  paper  as  the  information 
transfer  medium. 


5.  Select  troubleshooting  test  cases  to  represent  (a)  complex  task 
electronics  systems,  (b)  not-so-complex  task  but  complex  system  electronics 
(c)  mechanical  systems,  and  (d)  integrated  mechanical/complex-digital  elec- 
tronics systems. 


6.  Coordinate  R&D  efforts  with  NTIPP  program  for  design  of  optimum 
JPA  selection  methodology  and  procurement  of  JPA  technical  data  test  cases. 


7.  Consider  development  of  total  life  cycle  tradeoff  model  for  JPAs, 
training,  job  design,  technical  documentation,  maintenance  design,  and  equip- 
ment design. 


8.  Review  Human  Factors  Engineering  Integration  Project  of  NSRDC, 
Annapolis  and  NAVPERSRANDCEN  proposed  study  on  Communications  Network  for 
Personnel  R&D  Products  and  assess  applicability  to  total  JPA  system  implemen 
tat ion  model. 


9.  Determine  and  initiate  development  of  policies  and  procedures 
required  to  integrate  Navy  training  and  technical  data  for  major  weapon 
systems. 


10.  Consider  development  of  DoD-wide  JPA  technology  standards  and 


Methodological  Studies 


1.  Conduct  tradeoff  study  to  determine  most  cost-effective  combination 
of  JPA  and  job-based  training  for  developing  skills  in  "deductive"  troubleshoot' 
ing. 


2.  Develop  methodology  for  integrating  job-based  training  with  JPA 


technology 


3.  Perform  study  on  the  implications  of  JPA  technology  for  Job  design 
personnel  careers,  and  job  satisfaction. 


4.  Develop  a JPA-based  personnel  system  concept  integrated  across  per 
sonnel,  training,  and  job  design. 


5.  Develop  methodology  and  collect  data  for  costs  criteria  to  be  used 
in  tradeoffs  among  (a)  JPA  techniques,  (b)  JPAs,  training,  and  job  design,  and 
(c)  JPAs  and  automatic  test  equipment. 


Future  R&D 


1.  Initiate  research  and  development  efforts  to  fill  performance 
data  gaps  In  the  JPA  technology  base  and  to  explore  the  feasibility  of  new 
concepts  for.JPAs. 


2.  Develop  a scientific  and  technical  approach  for  Identifying, 
developing,  testing,  and/or  evaluating  principles  and  methodology  applicable 
to  the  implementation  of  advanced  technology  into  Navy  weapon  and  logistics 
support  systems. 
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INTRODUCTION 


Robert  E.  Blanchard 

Navy  Personnel  Research  and  Development  Center 
Problem  and  Background 

Currently,  the  Navy  (as  well  as  other  services)  is  faced  with  burgeoning 
personnel  costs  accompanied  by  decreasing  personnel  force  levels  and  declining 
entry-level  skills.  Maintaining  required  operational  readiness  levels  under 
strict  budget  limitations  is  an  ever-increasing  problem. 

A substantial  amount  of  data,  collected  over  the  past  20  years  by  the 
Army,  Navy,  Coast  Guard,  and  particularly  the  Air  Force,  suggest  that  job 
performance  aids  can  be  employed  to  enhance  performance  of  lesser-trained  or 
lesser-skilled  individuals,  and  to  reduce  training  and  maintenance  costs. 

For  example,  in  a study  of  low-cost  ownership  for  the  Army,  Shriver  and 
Hart  (1975)  estimated  that  savings  approaching  $1.7  billion  annually  could 
be  realized  in  Army  personnel,  training,  and  maintenance  costs  through  the  use 
of  JPAs.  In  a study  for  the  Advanced  Research  Projects  Agency,  Rowan  (1973) 
estimated  that  the  use  of  spares  in  electronic  maintenance  could  be  reduced  by 
30  percent  through  JPAs.  The  study  completed  for  the  Navy  by  Post  and  Brooks 
(1970)  showed  a 25  percent  reduction  in  aircraft  maintenance  waiting  time 
through  a change  in  maintenance  workload  permitted  through  use  of  JPAs. 

Although  potential  payoffs  in  performance  increments  and  cost  avoidance 
appear  to  be  readily  available  from  JPA  technology  for  the  most  part,  JPAs 
have  been  implemented  only  in  piece-meal  fashion  by  any  of  the  military  services. 
Recently,  the  Army  initiated  a single  program  involving  concurrent  development 
of  training  materials  and  job  performance  aids,  which  is  called  "Improved 
Technical  Documentation  and  Training"  (ITDT) . More  significantly,  the  program 
could  have  been  labeled  "Integrated  Technical  Documentation  and  Training," 
which  would  be  more  descriptive  of  the  impact  the  study  could  have  in  military 
personnel  and  training.  Conversely,  the  Navy  and  Air  Force  to  date  have  been 
reluctant  to  modify  current  policy  with  respect  to  integration  of  training 
design  and  job  performance  aids  and  have  chosen  to  seek  further  evidence  of 
JPA  payoffs  and  potential  impact  on  current  personnel  systems. 

Although  most  of  the  previous  research  shows  positive  results  with  use  of 
JPA  technology,  several  criticisms  have  been  raised.  Rowan  (1973),  for  example, 
was  critical  of  the  experimental  designs  and  statistical  analyses  used  in  many 
of  the  studies;  however,  he  agreed  with  the  conclusions  drawn  regarding  the 
performance-enhancing  and  cost-saving  properties  of  the  JPAs  tested.  Another 
complaint  is  that  there  have  been  few  studies  that  compare  one  JPA  to  another, 

. and  since  there  are  literally  hundreds  of  systems  that  claim  JPA-like  features, 

a program  manager  or  other  decision  maker  has  little  data  at  hand  to  aid  him 
in  selecting  a particular  JPA  approach. 
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Purpose 


NAVPERSRANDCEN  currently  is  conducting  a major  effort  concerning  the 
test  and  evaluation  of  job  performance  aids  per  se  and  their  integration 
with  current  Navy  personnel  and  training  systems.  Although  a thorough 
literature  review  and  critique  were  performed,  it  was  felt  that  full  under- 
standing of  the  current  state-of-the-art  with  respect  to  JPA  technique 
applicability  and  potential  negative  impact  effects  was  not  at  hand.  Further, 
there  were  insufficient  data  upon  which  to  base  the  always  critical  decisions 
regarding  maximum  payoff  for  R&D  time  and  funds  invested;  that  is,  what 
approach  do  we  take  to  get  the  most  of  our  time  and  money? 

Approach 

As  one  approach  toward  resolving  this  dilemma,  an  invitational  conference 
was  organized  to  which  a limited  number  of  nationally  recognized  experts 
within  the  field  of  JPAs  were  invited.  Each  was  asked  to  prepare  a technical 
paper  on  an  assigned  topic.  Informal  discussion  among  the  group  was  planned 
following  each  paper. 

One  major  area  addressed  concerned  defining  the  current  state-of-the-art 
in  JPA  technology,  particularly  as  regards  personnel  and  training  implications. 
There  is  a host  of  JPA  techniques  and  approaches  apparently  ready  and  available 
for  use.  However,  insufficient  guidelines  are  provided  for  selecting  the 
optimum  approach,  considering  such  factors  as  personnel  characteristics,  job 
tasks,  environment,  procurement  resources,  and  so  forth.  The  very  first  step, 
then,  that  of  selecting  optimum  JPA  approaches  for  demonstration,  was  not 
readily  available  as  part  of  the  JPA  technology  base.  Also,  issues  concerning 
the  potential  impact  on  personnel  and  training  from  JPA  applications  could  not 
be  addressed  directly  without  a better  definition  of  the  JPA  technology  base. 

A second  area  concerned  obtaining  a better  overall  conceptualization  of 
job  aiding;  problems  in  procuring,  producing  and  evaluating  JPAs;  and  major 
considerations  in  obtaining  acceptance  by  both  procurement  officials  and 
operational  users  to  ensure  JPA  utilization.  This  topic  was  intended  to  bring 
together  the  experiences  and  views  of  individuals  in  the  logistics,  technical 
data,  production,  and  research  and  development  communities. 

Finally,  conference  participants  were  asked  to  identify  and/or  recommend 
approaches  for  integrating  JPA  technology  into  the  Navy  personnel  and  training 
system  for  the  1980s.  Attention  here  was  also  devoted  to  the  various  types  of 
training  and  delivery  systems  that  would  be  involved.  Also,  perceived  prob- 
lems in  integrating  JPAs  into  existing  personnel  systems  were  noted,  partic- 
ularly those  concerning  career  structuring,  advancement  in  rank,  assignment 
procedures,  and  recruiting  and  retention.  It  would  seem  that,  to  the  extent 
the  objectives  of  integrated  JPAs  are  met,  the  military  services  will  be  that 
much  closer  to  realizing  the  full,  potential  benefits  of  JPA  technology. 
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SOME  PERSPECTIVES  ON  THE  JOB 
PERFORMANCE  AIDS  TECHNOLOGY  BASE 

John  J.  Collins 
Essex  Corporation 


Introduction 


It  appears  desirable  to  begin  our  discussion  of  the  JPA  technology  base 
with  a statement  of  the  principal  elements  of  that  base.  In  this  paper,  1 
will  be  including  theories,  principles,  methods  and  techniques,  data,  and 
applications  experience  related  to  JPA  development  and  utilization  when  I 
refer  to  the  technology  base.  In  this  context,  the  literature  on  job  per- 
formance aids  indicates  to  me  that  the  technology  base  is  still  very  much 
in  an  emerging  and  developing  stage.  It  seems  to  be  undergoing  a meta- 
morphosis at  this  time,  which  is  providing  an  opportunity  to  capitalize  on 
the  accomplishments  of  the  past  in  being  responsive  to  the  demands  of  the 
present  and  the  future.  It  is  within  this  general  premise  that  I wish  to 
direct  my  comments  today.  It  is  not  my  objective  to  try  to  address  all  of 
the  elements  of  the  technology  base  or  to  focus  on  specific  elements  in  a 
comprehensive  way.  Rather,  I will  try  to  provide  some  perspectives  on 
selected  developments  that  are  influencing  the  evolution  of  the  technology, 
to  examine  some  of  the  issues  relevant  to  the  purposes  of  this  symposium, 
and  to  suggest  some  research  and  development  needed  to  continue  progress  in 
JPA  development  and  applications. 

Some  General  Characteristics  of  JPA  Field 


First,  let  us  look  at  some  general  characteristics  of  the  field  of  job 
performance  aids.  Ayoub  and  his  colleagues  (Ayoub,  Cole,  Sakala,  & Smillie, 
1974)  have  reviewed  several  aspects  of  the  status  of  job  performance  aids 
development  and  utilization  and  provide  some  interesting  insights.  For 
example,  they  found  that  78  percent  of  authors  made  no  more  than  a single 
contribution  in  the  last  20  years.  They  conclude  that  this  clearly  indicates 
a small  rate  of  retention  of  JPA  researchers  and  practitioners  in  the  field. 
They  also  found  that  state-of-the-art  JPA  developments  have  centered  on  the 
efforts  of  single  individuals  rather  than  groups  of  individuals.  Another 
measure  they  used  is  the  number  and  distribution  of  publications.  From  their 
data,  they  concluded  that  the  growth  in  the  state-of-the-art  is  rather  slow. 
The  statistics  cited  are  that  only  23  percent  of  the  JPA  literature  appears 
in  scientific  and  technical  journals,  and  that  there  has  been  a drop  in  the 
number  of  publications  from  a high  of  40  in  1971  to  4 in  1974.  However,  a 
different  perspective  can  be  gained  by  further  analyses  of  their  data.  Group- 
ing the  data  by  decades,  we  find  that  the  averages  for  1950-1959,  1960-1969, 
and  1970-1974  were  4.7,  14.3  and  19.6  publications  per  year,  respectively. 

From  this  grouping  of  data,  one  might  well  conclude  that  the  rate  of  growth 
is  increasing  rather  than  decreasing.  Determining  whether  the  growth  rate 
is  slow,  moderate,  or  fast  requires  a more  definitive  analysis  and  collection 
of  relevant  measures  than  is  contained  in  this  study. 
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Major  Conceptual  and  Theoretical  Trends  In  Work  and  Work  Situations 

Next,  let  us  look  very  briefly  at  some  of  the  directions  and  emphases 
in  work  and  the  work  situation  developments  and  their  significance  for  JPA 
developments.  We  have  moved  during  the  past  100  years  through  a series  of 
conceptual  and  theoretical  developments  (Barrett  & Dambrot,  1975;  Barrett, 
Dambrot,  & Smith,  1975).  The  concern  during  the  last  century  was  on  physical 
exertion  and  how  much  physical  work  a man  can  perform  in  a stated  period  of 
time.  In  the  early  1900s,  this  concern  with  physical  capacity  was  extended 
to  include  simplification  of  the  task  and  the  "one  tfest  method"  to  perform 
each  individual  job.  Later,  factors  of  fatigue  and  rest  and  their  relation- 
ship to  productivity  were  emphasized.  From  that  orientation,  we  moved  to 
improvement  of  equipment,  tools,  work  space,  and  body  movement — the  in- 
dustrial engineering  approach.  These  early  models  assumed  that  productivity 
could  be  maximized  by  designing  jobs  for  the  persons  with  minimal  ability 
and  motivation.  Work  simplification  and  detailed  specifications  of  task 
requirements  were  believed  to  reduce  the  importance  of  ability  and  motiva- 
tional levels.  Other  assumptions  included  the  absence  of  any  consequential 
interactions  between  individual  attributes  and  work  characteristics  that 
would  influence  performance. 

The  emphasis  shifted  during  the  early  1940s  from  this  concern  with 
physical  work  to  mental  work;  that  is,  sensory,  perceptual,  and  decision- 
making abilities  required  in  complex  man-machine  systems.  Simultaneously, 
the  shift  from  the  work  simplification  and  work  specialization  emphasis  to 
work  motivation  concepts  took  place,  and  theoretical  and  empirical  studies 
relating  to  various  forms  of  motivation  increased.  The  basic  theoretical 
framework  for  much  of  this  emphasis  came  from  Maslow's  Hierarchical  Theory 
of  Motivation  and  Hertzberg's  theory  of  intrinsic-extrinsic  motivation,  the 
orientation  in  these  theories  again  being  to  minimize  interaction  between 
individual  abilities  and  attributes  and  job  structuring.  A sociological 
approach  focusing  on  groups  and  group  differences  and  interactions  with  jobs 
and  group  membership  also  appeared.  This  was  the  first  explicit  statement 
of  the  interaction  effects  of  cultural  values  but  only  at  the  group  level. 

Motivation  theory  has  provided  the  framework  for  much  of  the  recent  atten- 
tion to  individual  worker  needs,  motives,  and  perceptions.  Theories  of  job 
enrichment,  job  enlargement,  vertical  and  horizontal  loading,  etc.  consider 
factors  of  Job  content,  various  psychological  states,  and  productivity  that 
are  moderated  by  workers'  growth  need  strength.  Models  have  developed  with 
consideration  not  only  of  individual  perceptions  and  motives  but  also  ability 
to  perform  a Job  and  other  individual  attributes  as  they  interact  with  job 
design  and  performance.  Still  other  approaches  encompass  concepts  of  organiza- 
tional development  or  management  by  objectives.  F.arly  work  in  small  group 
research  has  brought  about  models  that  view  jobs  as  a complex  field  involving 
organizational,  technical,  and  personal  aspects.  This  theoretical  orientation 
focuses  on  the  relationship  among  technology,  organizational  structure,  and 
human  Interactions  in  work  groups,  and  on  the  goals  of  satisfying  both  organiza- 
tional task  requirements  and  human  requirements. 
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New  conceptual  approaches  to  job  structuring  that  have  developed  include: 

(1)  a behaviorally  descriptive  approach,  (2)  a task  requirements  approach 
using  behaviorally-oriented  dimensions,  (3)  an  approach  labeled  "task-qua- 
task"  that  focuses  on  the  physical  properties  of  the  task,  and  (4)  an 
abilities  requirements  approach.  New  methods  and  techniques  have  also 
been  developed  for  defining  and  measuring  tasks  in  a variety  of  ways.  These 
methodologies  include  a rational  approach  to  task  taxonomy,  observation 
and  rating  scales  by  experts,  ability  reference  tests,  and  quantification 
of  the  physical  properties  of  a task.  Especially  significant  are  the  renewed 
and  increasing  developments  in  the  areas  of  group  performance  and  productiv- 
ity. All  of  these  techniques  represent  capabilities  that  can  be  applied 
or  adapted  for  application  to  meet  various  requirements  for  JPA  developments. 

Briefly,  we  might  state  that  we  have  witnessed  the  development  of  a tech- 
nology base  in  work  and  work  environments  with  shifts  in  emphasis  from  the 
machine  to  the  human  element,  from  work  tools  and  methods  to  work  context  or 
the  work  itself,  and  from  management  goals  of  profit  and  loss  to  joint  organiza- 
tional and  human  goals.  In  the  process,  numerous  and  diverse  theories,  con- 
cepts, and  approaches  have  been  developed  and  are  being  tried.  Applications 
can  be  found  in  all  sections  of  the  society  including  the  military  establish- 
ments at  the  present  time.  Thought  is  also  being  given  to  the  future,  and 
an  interesting  and  original  set  of  papers  on  work  and  nonwork  in  the  year  2001 
was  recently  published  by  Dunnette  (1973). 

Recent  developments  and  emphases  in  work  design  and  work  environments 
on  the  quality  of  life  have  some  parallels  in  learning  theory.  For  example, 
the  experiential  learning  model  and  its  practical  counterpart,  the  action 
research  method,  has  its  foundation  also  in  Lewinian  theory  (Kolb  & Fry,  1975). 
The  underlying  concept  is  that  learning,  change,  and  growth  are  best  facilitated 
by  an  integrated  process  involving  immediate  experience,  validation  of  that 
experience,  modification  of  that  behavior,  and  the  choice  of  new  experiences. 
Without  presenting  an  in-depth  description  of  this  theory,  I would  point  out 
that  the  theory  (1)  provides  a framework  for  the  integration  of  the  cognitive 
and  socioerootional  perspectives  in  the  learning  process,  (2)  allows  for  the 
recognition  and  description  of  individual  differences  in  learning  styles  that 
shape  behavior  not  only  in  traditional  educational  settings  but  also  in  the 
individual's  basic  mode  of  adaptation  to  the  world  around  him,  and  (3)  describes 
the  life  cycle  of  human  development  through  various  stages  with  their  dif- 
ferentiating learning  styles. 

These  two  brief  looks  at  major  conceptual  and  methodological  developments 
in  the  work  and  learning  areas  suggest  to  me  some  general  criteria  or  require- 
ments for  the  further  developments  of  job  performance  aid  technology.  Perhaps 
it  would  be  useful  at  this  point  to  pose  these  in  the  form  of  questions  that 
might  be  further  examined  in  this  paper  and  in  our  discussions — questions  that 
appear  helpful  in  establishing  a capacity  to  store  information  for  later 
retrieval  in  connection  with  the  performance  of  a job.  The  aid  facilitates 
performance  by  reducing  the  memory  and  possibly  training  requirements  imposed 
upon  the  performer.  Joyce  (1975)  has  advocated  the  integration  of  a JPA/Job- 
oriented  training  approach.  Folley  (1961)  also  defined  JPAs  as  aiding  in 
information  handling. 


Wulff  and  Berry  (1962)  stress  the  guidance  aspects  of  job  aids  rather 
than  their  use  as  reference  materials.  That  is,  an  aid  is  something  that 
guides  an  individual's  performance  so  as  to  do  something  he  was  not  previ- 
ously able  to  do.  To  these  authors,  achieving  stimulus  control  is  essential 
in  good  job  aid  design.  Rowan  (1973)  has  emphasized  the  guidance  role  of 
JPAs  and  believes  the  aids  are  people  rather  than  equipment-oriented. 

Chalupsky  and  Kopf  (1967)  address  the  information  content  and  instruction 
aspects  of  aids  and  the  facilitating  effect  on  performance  by  relieving 
certain  job  demands.  Lewis  and  Cook  (1969)  provide  a different  information 
approach  entitled  "Theory  of  Telling,"  a unidirectional  communication  between 
man  (user)  and  a JPA.  Since  there  is  no  feedback  from  the  user,  they  point 
out  that  the  design  of  JPAs  must  be  responsive  to  a model  of  the  user's 
abilities,  limitations,  and  processing  functions.  Pictorial  and  linguistic 
channels  are  the  primary  means  of  telling.  Algorithms  in  the  form  of  ques- 
tion-and-answer  flow  charts  and  question  lists  are  used  as  improvements  over 
traditional  prose.  Wolfe  (1975)  at  NAVPERSRANDCEN  has  been  developing  a related 
capability  as  an  aid  for  independent  study  through  automatic  question  generation 
(AUTOQUEST) . 

Another  perspective  on  the  technology  base  can  be  gained  from  a look  at 
two  analyses  of  job  performance  aid  developments  by  Main,  Harrigan,  and 
Hooprich  in  1971  and  Rowan  in  1973.  Main  et  al.  have  been  interested  in 
identifying  methods  for  furthering  job  aid  development  and  especially  imple- 
mentation (Main  et  al.,  1971;  Main,  1974).  In  the  1971  study,  they  point  out, 
from  their  review  of  the  literature,  three  limiting  influences  on  JPA  develop- 
ments and  utilization.  The  first  is  a lack  of  conceptual  development  as  a 
class  of  devices;  the  second  is  the  narrowness  of  research;  and  the  third  is 
the  failure  to  establish  effective  methods  of  implementation.  Main's  recent 
report  (1974)  used  a mail-out  questionnaire  approach  to  determine  the  adequacy 
of  current  uses  of  JPAs,  to  identify  the  requirements  for  increased  utilization 
of  aids,  and  to  elicit  suggestions  for  JPA  applications.  He  concluded  that 
there  is  a recognized  need  for  expansion  of  the  implementation  of  JPAs  and 
general  dissatisfaction  with  the  distribution  of  available  devices.  Main  also 
found  that  the  three  types  of  assistance  desired  most  frequently  by  respondents 
were  in  (1)  equipment  operation,  (2)  reducing  time  requirements,  and  (3) 
utilizing  Inexperienced  personnel.  These  latter  findings  provide  insight  into 
both  satisfactions  and  expectations  of  users,  approaches  in  job  aid  development. 

A final  example  of  an  analysis  of  job  performance  aid  developments  is  the 
study  by  Rowan  (1973).  His  report  presents  the  results  of  a comparison  of 
innovative  job  aids  versus  conventional  documentation.  The  concepts  examined 
were  developed  between  1958-1972  and  included  aids  such  as  FORECAST,  JOBTRAIN, 
SIMMS,  PIMO,  FPJPA,  SAFEGUARD,  and  nontroubleshooting  JPAs.  While  citing 
concerns  about  some  of  the  evaluation  approaches  used.  Rowan  points  out  the 
many  positive  results  reported  in  the  literature  in  the  development  and  applica- 
tion of  these  performance  aids.  The  results  include  large  reductions  in  train- 
ing time;  significant  increases  in  performance  as  measured  by  number  of  faults 
located,  time  to  complete  tasks,  number  of  errors,  etc.;  and  effective  use  of 
performance  aids  by  inexperienced  personnel.  Other  positive  assessments  of 
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new  concepts  in  maintenance  performance  aids  are  contained  in  the  proceedings 
of  the  1975  Naval  Training  Equipment  Center  symposium  on  this  subject  (King 
& Duva,  1975,  U.S.  Navy,  1976)  and  the  Second  Annual  Job  Information  Pre- 
sentation Conference  (Rowan  & Genovese,  1976).  In  the  summary  of  the  King 
and  Duva  report  King  states: 

In  summary,  it  is  apparent  from  papers  presented  in  this 
volume  and  in  the  discussions  held  during  the  3-day  workshop 
that  the  technologies  required  for  significant  improvements 
in  maintenance  are  ready  and  available.  What  seems  to  be  needed 
are  adequate  demonstrations  to  the  user  communities  of  those  cost 
reductions  and  training  benefits  that  are  likely  to  accrue  from 
use  of  these  newer  technologies. 

In  discussing  the  technology  base  definitions,  concepts,  and  methodologies 
in  the  context  of  an  analytical  framework,  there  is  the  danger  in  a short 
paper  of  this  kind  that  the  more  positive  aspects  might  not  be  given  adequate 
representation.  There  is  a large  body  of  information,  well  known  to  all  of 
the  participants  in  this  symposium,  in  areas  such  as  planning  for  JPAs, 
principles  and  techniques  of  development,  design  characteristics  and  speci- 
fications, classification  and  categorizing  systems,  production  and  implemen- 
tation, and  test  and  evaluation,  including  criteria  of  cost,  accuracy,  and 
time.  This  technology  base  supports  the  well-documented  objective  of  all 
JPA  approaches  of  providing  in  a single  package  complete  and  current  infor- 
mation to  the  user  with  minimal  need  for  cross-referencing  and  retention. 

Suggested  Areas  of  Research  and  Development 

I would  suggest,  however,  that  there  are  several  research  and  develop- 
ment problems  needing  resolutions  to  improve  this  technology  base  as  well 
as  to  meet  the  demands  of  broader  objectives  that  have  been  proposed  for 
the  use  of  JPAs.  These  objectives  require  new  capabilities  that,  in  some 
instances,  are  conceptually  and  methodologically  more  complex  and  more 
sophisticated  than  the  existing  technology  base  will  support.  For  example, 
the  objective  of  fully  integrating  job  performance  aids  into  the  mainstream 
of  manpower,  personnel,  and  training  systems  requires  not  only  refinements 
and  improvements  of  present  capabilities  but  also  additions  to  the  technology 
base  that  are  consistent  with  supportive  theoretical,  methodological,  and 
operational  developments.  This  objective  requires  a dual  strategy  of  improving 
the  known  and  searching  for  the  unknown — or  at  least  the  yet  undeveloped. 

With  this  strategy  suggestion  in  mind,  let  me  end  this  brief  presentation 
by  enumerating  some  JPA  capabilities  requiring  research  and  development. 

These  areas  of  needed  research  are  presented  without  comment  for  possible 
discussion  during  the  symposium. 

1.  A general  method  for  assessing  and  evaluating  JPAs. 

2.  An  improved  data  base  developed  through  experimental  investigations 
and  field  data  collection  for  designing  and  producing  JPAs. 
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3.  Specific  quantitatively-oriented  rules  and  guidelines  for  develop- 
ing and  producing  JPAs. 

4.  Improved  distribution  and  delivery  systems  for  available  and  future 
JPAs. 


5.  Techniques  for  collecting  and  processing  detailed  requirements  in- 
formation for  JPAs. 


6.  Cost-effective  approaches  to  JPAs. 

7.  JPAs  keyed  to  group  performance  requirements. 

8.  Alternative  uses  of  JPAs,  e.g.,  for  feedback  and  system  updating. 

9.  Principles  and  techniques  for  designing  JPAs  to  fit  previous  learning, 
job,  and  life  style. 

10.  Approaches  for  integrating  JPAs  early  Into  the  system  planning  and 
acquisition  cycle. 

11.  Approaches  suited  to  the  more  complex  dimensions  of  job  performance 
requirements. 

12.  Decision  criteria  and  tradeoff  techniques  for  determining  optimum 
JPA  uses. 


SELECTION  OF  FORMATS  AND  MEDIA  FOR 
PRESENTING  MAINTENANCE  INFORMATION 

Theodore  Post 
BioTechnology,  Inc. 

Problem/Current  Situation 

To  convey  the  idea  of  the  format  and  medium  selection  problem.  Figure  1 
lists  some  hypothetical  hardware  systems  and  a few  available  specifications. 

The  problem  is  to  choose  one  or  more  of  the  "TM  Specs"  on  the  right  to  meet 
the  maintenance  information  needs  of  the  systems  on  the  left.  At  present, 
the  Navy  has  no  consistent  method  for  guiding  component  engineers  or  Program 
Managers  in  making  this  choice.  As  a consequence,  the  decision  maker  usually 
chooses:  (1)  the  safe  spec  (15071G),  (2)  the  spec  he  used  last  time,  or  (3) 
the  spec  that  was  touted  by  the  last  visiting  salesman. 

I contend  that  this  inconsistent  approach  to  choosing  technical  manual 
(TM)  information  can  have  adverse  effects  on  TM  usage.  This  type  of  selection 
process  can  have  unfavorable  effects  on  costs,  training  effectiveness,  manpower 
utilization,  and  system  effectiveness.  However,  a number  of  these  TM  problems 
can  be  avoided  during  the  selection  process. 

One  unfavorable  effect  is  that  TMs  are  not  getting  the  utilization  we 
expect.  The  lack  of  use  is  evident  in  the  data  presented  in  Table  1.  (Only 
columns  1,  3,  and  5 are  relevant  to  the  point  being  made  here.)  The  amount 
of  man  time  (MT)  spent  using  the  "tech  orders"  (column  3)  compared  to  elapsed 
time  (column  1)  varies  from  0 to  24  percent,  the  mean  being  3.5  percent.  Scant 
use  of  TM  information  obviously.  However,  the  number  of  minutes  of  procedural 
problems  (column  5)  is  also  quite  small.  This  implies  that  maintenance  actions 
(MAs)  are  completed  acceptably.  Maybe  underutilization  of  TMs  is  not  really  a 
problem. 

Table  2 provides  information  to  address  the  quality  of  performance  question. 
The  table  presents  the  results  of  a survey  of  workcenter  chiefs  who  were  asked 
which  of  their  men  could  perform  the  workcenter 's  MA  (for  F4Js).  Only  non- 
troubleshooting MAs  were  included  and,  even  so,  the  average  man  could  perform 
only  65  percent  of  the  necessary  MAs.  If  troubleshooting  MAs  were  included, 
the  figure  would  undoubtedly  drop  below  50  percent.  These  same  chiefs  were 
asked  earlier  what  they  thought  contributed  to  their  men's  poor  performance 
capability  and  the  vast  majority  said  poor  retention.  Thus,  there  is  some 
evidence  that  TMs  receive  little  use  and  personnel  capability  is  suspect. 

The  Warrants  Approach 

Objectives 

The  warrants  concept  hypothesizes  that  TMs  are  not  used  because  of 
mismatches  between  TM  characteristics  and  conditions  within  the  system  being 
supported.  Thus,  TM  potential  to  improve  technician  performance  and  maintenance 
effectiveness  is  not  maximized. 


Table  1. 

Maintenance  Man  Time,  by  Maintenance  Actions,  (in  minutes) 
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The  conditions  under  which  MAs  are  performed  change,  sometimes  even 
within  one  system,  and,  to  provide  best  support,  the  TM  must  be  compatible 
with  these  conditions.  For  example,  some  maintenance  requires  that  the 
prime  system  be  "down,"  not  operationally  ready.  Other  maintenance  has  no 
immediate  effect  on  the  readiness  of  the  prime  system.  The  warrants  approach 
attempts  to  consider  differences  such  as  these  in  selecting  formats  and  media 
for  presenting  a system's  maintenance  information. 

Figure  2 depicts  in  a general  way  the  range  of  outcomes  that  can  be 
expected  from  a good  and  a poor  match  between  TM  characteristics  and  system 
conditions.  The  rays  from  "good  match"  and  "poor  match"  flare  out  to  in- 
dicate that,  even  with  the  best  or  worst  of  matches,  there  will  be  variation 
in  both  usage  and  performance  effectiveness,  due  to  factors  such  as  morale, 
type  of  mission,  and  hardware  design.  The  point  is  that,  with  a poor  natch, 
highest  usage  levels  are  not  likely  to  be  attained,  and  task  performance  and 
maintenance  effectiveness  will  be  short  of  their  potential.  The  objective 
then  is  to  match  the  characteristics  of  TMs  to  relevant  conditions  of  the 
maintenance  situation. 

The  number  of  TM  options  available  for  application  to  various  systems 
is  bewildering  (Figure  3).  Many  of  these  options  have  been  tested  and  indicate 
promising  results;  others  have  not  been  tested  but  are  supported  by  research 
principles;  and  still  others  are  merely  ideas  of  interested  persons.  With 
few  exceptions,  notably  those  listed  in  Figure  1,  these  TM  options  have  one 
thing  in  common — none  of  the  services  have  adopted  them  for  any  but  experi- 
mental applications.  When  examined  closely  for  meaningful  differences,  most 
of  the  TM  options  fall  into  relatively  few  categories.  In  fact,  a beginning 
can  be  made  by  considering  TM  options  as  variations  of  (1)  proceduralized 
or  (2)  deductive  aid  formats. 

It  should  be  noted  that  the  TM  options  vary  in  both  format  and  medium 
(viz.,  recording  mode,  display  mode,  and  access).  The  warrants  concept  con- 
tends that  formats  and  media  can  be  selected  for  each  system  on  the  basis  of 
conditions  within  that  system. 

System  Conditions  and  Methodology 

Table  3 lists  the  current  set  of  system  conditions  considered  useful 
in  selecting  formats  and  media.  Inasmuch  as  selection  of  TM  options  occurs 
early  in  the  system  development  process,  it  is  necessary  to  have  system  con- 
ditions that  are  identifiable  early,  and  that  are  reasonably  easy  for  a Pro- 
gram Manager  to  define.  A major  objective  of  the  warrants  approach  is  that 
we  want  to  affect  TMs  when  they  are  selected  by  a Program  Manager,  early  in 
system  development. 

In  addition  to  their  ability  to  suggest  TM  formats  and  media,  each 
condition  affects  one  or  more  aspect  of  system  performance  (see  Table  3).  For 
example,  if  the  TM  format  is  selected  to  accommodate  a certain  rate  of  person- 
nel turnover,  the  effect  anticipated  is  that  personnel  cost-effectiveness  will 
improve,  as  the  technician  will  then  have  technical  information  matched  to  his 
performance  capabilities.  As  shown  in  Figure  4,  personnel  turnover  can  occur 
in  either  of  two  states,  each  of  which  suggests  a particular  type  of  format. 

The  format-medium  indications  of  a single  condition  are  not  necessarily  con- 
clusive. However,  the  case  becomes  stronger  if  enough  other  conditions  relat- 
ing to  the  same  TM  decision  are  in  agreement;  or,  if  the  other  conditions  dis- 
agree, tradeoffs  are  set  up. 
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Figure  2.  Solution  Concept. 
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Figure  3.  Some  Available  Format-Media  Choices. 
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The  preliminary  version  of  the  warrants  process  relies  on  the 
19  conditions  of  Table  3 to  help  select  formats  and  media.  These  condi- 
tions are  considered  in  a five-step  methodology  summarized  in  Figure  5. 

Step  1 of  the  process  is  a review  of  the  Task  Identification  Matrix  (TIM). 
The  purpose  of  this  review  is  to  group  tasks  into  sets  that  will  economize 
on  the  number  of  format  and  medium  decisions  that  have  to  be  made.  In  this 
step  also,  information  is  gathered  to  defined  the  conditions  of  the  system 
under  development. 

Format  selections  are  made  in  steps  2,  3,  and  4 while  step  5 selects 
the  medium.  In  steps  3 and  4,  one  of  several  alternative  formats  is  chosen 
to  support  the  performance  of  either  remove-replace  tasks  (MAs)  or  trouble- 
shooting tasks  (MAs).  The  choice  is  based  on  the  conditions  referred  to 
earlier  and  the  cost  of  preparing  the  formats. 

Tne  special  case  maintenance  actions  identified  in  step  2 are 
unique  types  of  tasks  where  it  appears  that  one  format  is  the  most  effec- 
tive regardless  of  the  conditions  under  which  the  task  is  performed. 

Flow  Chart  of  the  Warrants  Process 


The  Figure  6 flow  chart  illustrates  the  decisions  to  be  made  in 
selecting  formats  to  support  troubleshooting  performance.  The  first  choice 
in  the  chart  uses  two  hardware  conditions — subordination  (high  or  low), 
and  diagnostic  technique  (internal  or  external) — to  form  four  possible  com- 
binations. The  second  choice  considers  whether  supervision  is  available  to 
support  technician  performance.  The  first  three  system  conditions  are  in- 
tended to  define  task  performance  complexity,  a major  indicator  of  the  type 
of  format  needed.  The  next  condition  deals  with  personnel  turnover,  while 
the  last  two  conditions  consider  the  effects  of  maintenance  demands  and 
system  criticality. 

The  heavy  outline  of  Figure  6 shows  a hypothetical  situation  in 
which  the  first  three  conditions  show  that:  an  internal  diagnostic  tech- 

nique is  required,  many  hardware  units  are  involved  in  the  diagnosis,  and 
no  supervision  is  available.  This  combination  of  conditions  is  felt  to 
represent  the  most  complex  task  performance  possible.  This  tends  to  suggest 
a proceduralized  format  approach.  In  the  next  consideration,  personnel  turn 
over  is  shown  as  "high,"  meaning  there  will  be  relatively  few  experienced 
technicians  available.  This  consideration  adds  strength  to  the  argument 
for  use  of  a proceduralized  format  for  MAs  to  be  performed  under  these  con- 
ditions. 


Continuing  to  refer  to  Figure  6,  the  heavy  line  shows  that  mainten- 
ance demands  are  "batch"  (i.e. , a large  number  of  MAs  occur  in  near  simulta- 
neous fashion) . The  result  of  considering  readiness  impact  indicates  that 
prime  systems  are  down  while  this  maintenance  is  being  performed.  The  need 
for  the  proceduralized  format  is  at  its  highest,  under  these  conditions  (i.e 
the  proceduralized  format  has  the  best  chance  of  allowing  the  inexperienced 
technicians  to  cope  with  the  complex  MAs  without  incurring  backlogs  that, 
in  turn,  cause  readiness  problems).  All  other  possible  condition  combina- 
tions can  be  followed  through  to  see  how  they  become  somewhat  less  critical. 


Complexity 


. Format  Selection  Flow  Chart. 


Conceptually,  the  warrants  approach  attempts  to  structure  the 
process  so  that  the  decision  maker  can  base  his  choice  on  the  type  of  com- 
parisons shown  in  Figure  7.  The  need  is  to  match  the  performance-fostering 
power  of  the  TM  format  to  the  criticality  of  the  maintenance-personnel  situa- 
tion in  which  the  TM  will  be  used.  There  is  no  point  in  spending  money  to 
buy  unneeded  performance  capability  (Figure  7,  "excess").  Conversely, 
there  is  no  point  in  saving  money  on  TMs  if  performance  capability  of  the 
format  falls  short  of  the  need  (Figure  7,  "shortfall"). 


Mismatch 


JPA 

PERFORMANCE 

CAPABILITY 

CRITICALITY 

EXCESS 


SHORTFALL 


DEDUCTIVE 

PERFORMANCE 

CAPABILITY 


Figure  7.  Mismatch  of  I’M  to  System  Criticality. 


Demonstrating  the  concept  requires  actual  data  that  ranks  the 
relative  strength  of  formats  under  various  task  complexities  (the  first 
three  choices  of  Figure  6).  Figure  8 presents  a hypothetical  plot  of 
troubleshooting  performance  on  tasks  of  four  complexity  levels.  Mote  that 
the  bottom  scale  of  Figure  8 (MA  characteristics)  is  not  a continuum  but 
represents  roughly  decreasing  levels  of  task  complexity  from  left  to  right. 
Several  assumptions  are  indicated  in  this  figure: 

1.  Performance  by  experienced  technicians  (open  symbols)  will 
be  better  than  by  inexperienced  (solid  symbols). 

2.  Performance  with  supervision  (O  ft  will  be  better  than 
without(CJ  & A)  . 

3.  Performance  will  improve  as  task  complexity  decreases. 

4.  There  may  be  interactions,  such  as  inexperienced  technicians 
with  supervision  (^)  performing  much  better  when  the  task  involves  low 
subordination  and  external  diagnostic  technique.  The  shaded  region  at 
the  top  of  the  figure  indicates  the  expected  range  of  performance  with 
proceduralized  job  performance  aids. 


21 


DRMANCE 


Cost  Considerations 


To  indicate  how  the  cost  considerations  fit  into  the  above  picture, 
turn  for  a moment  to  Table  4.  The  amounts  listed  were  arrived  at  using 
cost  guidelines  which  are  based  largely  on  subordination  (see  Post,  Price, 

& Diffley,  1976).  Costs  for  two  proceduralized  and  two  deductive  formats 
are  shown  for  three  levels  of  subordination. 

Return  to  Figure  8 and  consider  the  ID/HS  condition  (internal 
diagnostic/high  subordination).  Increasing  performance  above  the  "accep- 
table" line  will  apparently  require  fully  or  partially  proceduralized  JPAs. 
Performance  quality  alone,  however,  is  not  the  sole  criterion  for  this 
decision.  In  this  example,  if  there  is  no  readiness  impact,  maintenance 
demands  are  single  event,  and  turnover  is  low,  the  need  for  a proceduralized 
approach  softens.  Cost  of  preparing  information  in  the  alternative  formats 
should  also  be  considered  in  this  decision.  If  we  assume  a 1 to  12  subordina- 
tion, we  can  see  that  there  is  a $2300  difference  (per  MA)  between  the  fully 
proceduralized  and  simple  logic  formats.  This  large  cost  difference  may 
combine  with  the  relatively  mild  personnel  and  maintenance  conditions  to 
suggest  the  simple  logic  diagram  format.  The  warrants  methodology  leaves 
the  final  choice  to  the  Program  Manager. 

As  another  example,  refer  again  to  Figure  8 and  consider  the  ED/LS 
condition  (external  diagnostic/low  subordination).  As  indicated,  perfor- 
mance with  the  deductive  aid  is  good  for  three  of  the  four  personnel/ task 
conditions.  Referring  to  Table  4 and  assuming  a 1 to  4 subordination, 
the  decision  not  to  use  a fully  proceduralized  format  in  this  case  could 
save  $900  ($1100  - 200  = $900).  Again,  the  decision  is  not  likely  to  be 
this  straightforward.  There  may  be  compelling  reasons  for  using  proce- 
duralized aids  even  though  performance  is  expected  to  be  good  with  deduc- 
tive aids.  Personnel  and  maintenance  conditions  must  also  be  considered. 

For  example,  if  personnel  turnover  is  high  and  maintenance  actions  occur 
in  batches,  there  may  not  be  enough  qualified,  experienced  technicians 
to  cope  with  the  workload  under  a deductive  aid  situation.  A lack  of 
supervision  could  detract  further  from  technician  performance  capability, 
while  a prime  system  involvement  would  heighten  the  consequences  of  this 
lack.  Thus,  even  though  the  deductive  aid  form  is  suggested  by  the  hypothe- 
tical data  of  Figure  8 and  by  its  relatively  low  preparation  cost,  the 
conditions  of  personnel  turnover,  maintenance  demands,  and  readiness  impact 
may  show  one  of  the  proceduralized  aid  forms  as  the  most  effective  in  a 
life  cycle  sense. 

Operational  Model 

Admittedly,  all  of  these  projections  are  theoretical.  The  costs, 
the  task  performance  levels,  and  the  relationships  between  system  and  per- 
sonnel conditions  are  all  subject  to  experimental  verification.  Despite 
the  lack  of  direct  empirical  evidence,  the  approach  seems  credible  and 
practical. 
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To  add  to  this  credibility,  the  following  text  will  use  the 
warrants  concepts  to  interpret  some  field  test  data  that  has  been  extended 
by  using  a simple  model  to  represent  the  operation  of  a maintenance  work- 
center  (Post  & Brooks,  1970).  Table  5 shows  the  results  of  a field  test 
that  compared  the  performance-fostering  capability  of  job  guides  with  the 
format  included  in  a conventional  manual  (viz.,  partially  proceduralized) . 


Table  5. 


Performance  Quality  with  Conventional 
Vs.  Pictorialized  (Job  Guides)  Presentations 


FIRST  TRIAL  DATA  - % SATISFACTORY  PERFORMANCE 

Experienced 

Inexperienced 

Average 

Existing  Procedures 

74% 

63% 

68% 

Experienced 

Inexperienced 

Average 

Job  Guides 

100% 

94% 

97% 

The  workcenter  model  included  a batch  demand  situation  whose  size 
was  based  on  typical  missions  and  equipment  failure  rates  as  defined  by 
3M  data.  The  ratio  of  experienced  to  inexperienced  technicians  conformed 
to  actual  (high  turnover)  conditions;  personnel  assignment  policy  was  based 
on  current  practice  which  reflected  poor  performance  capabilities.  Two  runs 
were  made  with  this  model:  one  with  the  performance-fostering  power  of  the 

conventional  manual  and  one  with  the  performance-fostering  power  of  the  job 
guide.  Manpower  utilization  and  maintenance  queue  comparisons  resulting 
from  these  runs  are  shown  in  Table  6.  These  comparisons  and  the  following 
points  illustrate  how  the  format  selection  process  might  work  when  fully 
developed.  First,  the  tasks  of  the  field  test  were  chosen  by  workcenter 
chiefs  to  be  those  to  which  inexperienced  technicians  were  not  assigned,  but 
if  such  assignments  could  be  made,  the  workcenter  would  function  more  effec- 
tively. Second,  the  protocol  of  the  field  test  did  not  allow  the  subjects 
to  seek  help  from  their  peers  or  supervisors.  On  these  bases,  these  tasks 
were  given  the  highest  complexity  rating  available,  comparable  to  no  super- 
vision, internal  diagnostic  technique,  and  high  subordination. 
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Tahir  6. 

Potential  Manpower  Utilization  Benefits  of  the  Job  Guide  Concept 


Maintenance  Variable 

Current 

Practice 

Job  Guides 

Percent  Change 
and  Comments 

Experienced  Technicians 

Available  for  complex  work 

32% 

49% 

52%  increase 

Required  for  non-troubleshooting  work 

68% 

51% 

25%  decrease 

Inexperienced  Technicians 

Full  responsibility  for  non-trouble- 

0% 

83%* 

20  of  24  man*  hours  available  dur- 

shooting  maintenance 

Assisting  and  observing  experienced 

technicians 

71% 

17% 

ing  12-hour  shift 

^k  Available,  but  not  required  by  the 

maintenance  workload 

29% 

0% 

^k  Time  awaiting  maintenance  for  lack  of 

6 hrs. 

4.5 

25%  decrease 

technicians 

hrs. 

Number  of  maintenance  actions  failing 
quality  assurance  check 

4 

i 

75%  decrease 

The  work  forces  involved  in  the  experiment  were  experiencing  high 
turnover,  meaning  that  there  were  relatively  few  experienced  technicians 
available  to  process  the  incoming  workload.  The  model  assumed  a 16-man 
work  force,  eight  of  whom  might  be  available  for  unscheduled  maintenance. 
These  eight  technicians  were  reduced  to  five  to  represent  dayshift,  non- 
troubleshooting; and  three  of  the  five  were  considered  experienced,  a very 
conservative  reflection  of  the  high  turnover  situation. 

In  the  job-guide  run  of  the  model,  inexperienced  technicians  were 
allowed  to  perform  any  tasks  while  in  the  conventional  run  and  their  use 
was  limited  to  assisting  an  experienced  technician  in  a two-man  task  or 
to  observing  an  experienced  technician  performing  a one-man  task. 

Moving  into  the  maintenance  conditions  portion  of  the  warrants 
process,  the  model  dealt  with  system- installed  tasks  (e.g.,  removing 
a faulty  component  from  the  prime  system,  which  means  the  prime  system  is 
out  of  commission  during  the  performance  of  the  tasks),  and  with  batch 
arrivals  of  maintenance  tasks.  Figure  9 shows  the  particulars  of  the 
batch  situation. 

This  combination  of  task,  personnel,  and  maintenance  conditions 
constitutes  the  most  severe  requirements  the  warrants  process  can  generate. 
In  this  situation,  we  might  assume  that  the  fully  proceduralized  job  aid 
costs  some  percentage  more  than  its  partially  proceduralized  counterpart 
and  that  this  premium  buys  the  personnel  and  maintenance  improvements 
shown. 
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MAINTENANCE  WORKLOAD 


co 

z 

o 


i— 

UJ 

o 

CD 

* 

< 

z 

- 

< 

- 

Cd 

UJ 

z 

CO 

LU 

CD 

Ul 

- 

z 

h- 

z 

(- 

CO 

o 

2Z 

< 

z 

DC 

* 1 

LU 

z 

1 — 1 

ZD 

m 

1 — 

CO 

UJ 

< 

O 

Cd 

c_> 

1- 

X 

31 

=3 

<c 

DZ 

z 

- 

o 

cc 

NH 

CD 

>- 

(— - 

zr 

LU 

CD 

g 

i — 1 

<c 

DZ 

CO 

CD 

co 

l — 

CD 

* — i 

DZ 

<c 

• 

t— 4 

LA 

Z 

QC 

i 

CO 

CD 

1— « 

UJ 

u_ 

>- — i 

LU 

cn 

CO 

z 

D_ 

1 

1— 

■ — i 

- 

►H 

LA 

LI— 

i 

LU 

1 — 

< 

OO 

- 

t— i 

LA 

~r~ 

U. 

X 

LU 

i — i 

DZ 

g 

OO 

\— 

V— • 

UJ 

ZC 

IU 

i — 1 

X 

or 

CO 

CO 

<C 

C/3 

UJ 

LU 

■=r 

i — i 

Ul 

r 

- - 

>- 

X 

•a: 

33 

sr 

Q 

CD 

CD 

< 

i- 

• 

1 

CO 

o 

LU 

LU 

CD 

CQ 

ZT 

Cd 

<C 

1— 

§ 

■% 

O 

tx 

o 

u_ 

<C 

UJ 

X 

3 

i 

to 

NQ 

Cd 

CD 

X 

CD 

u 

to 

1— 

LU 

U— 

1- 

< 

O 

CD 

1 

Z 

~^rw' 

Cd 

Ul 

>- 

zr 

ZD 

UJ 

LU 

LU 

CD 

a; 

CQ 

<C 

CO 

LU 

CO 

Q_ 

z 

z 

o 

LU 

~~r 

•— « 

< 

§ 

1=1 

CO 

Cd 

h~ 

CO 

LU 

ct: 

»— « 

e 

LlJ 

~ 3 

LU 

CO 

3 

CD 

1— 

•=r 

a 

CQ 

o 

LU 

CD 

4—4 

c 

s 

i 

CO 

1—* 

DZ 

z 

CQ 

u_ 

LU 

CO 

1 — 

1— 

CD 

X 

UJ 

Cd 

LU 

CO 

UJ 

CD 

LA 

Z3 

> 

<r 

U— 

x 

Ul 

UJ 

1 

1 

1 

CD 

DC 

1- 

- 

CO 

■=r 

i — » 

CD 

LU 

o 

CO 

<c 

> 

CO 

< — 1 

li- 

CD 

CQ 

CD 

co 

LI— 

CM 

ar 

UJ 

O 

.s° 

<x 

<c 

5 

i— — i 

<r 

UJ 

o 

3 

o 

zc 

>- 

Q- 

z 

1 

co 

£ 

LU 

1 

Ul 

Z 

DZ 

DZ 

m 

h- 

LU 

LU 

h-4 

< 

Cd 

CO 

1 1 

— > 

V— 

DC 

a: 

X 

o 

1 

>— « 

<c 

< 

Ul 

~^g 

ZD 

>- 

i— i 

<c 

sz 

CL 

CM 

3 

1 

<C 

1 1 

X 

X 

UJ 

LU 

U— 

X 

o 

UJ 

UJ 

CO 

1 — 

1 

CD 

4—4 

or 

zc 

<=C 

Cd 

X 

r— 1 

< 

CO 

sz 

CO 

h— 

Q_ 

~£. 

z: 

<c 

1— 1 

<c 

CD 

Q_ 

UJ 

CO 

£ 

UJ 

X 

LU 

1— 

<r 

U 

DC 

z 

o 

O 

4-^ 

o 

OC 

3 

1—4 

*— l 

Q_ 

LA 

a 

<c 

Q_ 

i — 1 

Ul 

O 

<c 

* 

or 

< 

I 


In  summary,  our  objectives  in  developing  the  warrants  concept 
are  as  follows: 

1.  To  make  the  selection  of  formats  and  media  more  systematic. 

2.  To  match  the  cost  and  performance-fostering  capability  of 
formats  to  the  needs  of  the  system  being  supported. 

3.  To  improve  level  of  TM  usage  (acceptance)  by  tailoring  formats 
to  the  needs  of  the  users. 

4.  To  justify  premium  costs  of  new  formats  in  terms  of  maintenance 
or  personnel  benefits. 

5.  To  consider  task,  maintenance,  and  personnel  conditions  to  arrive 
at  a mix  of  aid  forms  required  by  the  system  being  supported. 

What  Next? 

We  are  hoping  to  collect  real  data  to  replace  the  hypothetical  and  to 
prepare  more  specific  guidelines  for  the  Program  Manager.  Ultimately,  we 
would  like  to  have  a tradeoff  model  using  weighted  system  conditions  and 
cost  algorithms.  We  don't  envision  this  model  as  ever  being  an  "automatic" 
decision  maker.  We  do  envision  a heuristic  model  that  could  be  easily  and 
reliably  applied  by  the  decision  maker — the  Program  Manager. 
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PROBLEMS  IN  PROCURING,  PRODUCING,  AND  EVALUATING  JPAs 


Frank  Johnson 
REM  Company 

Procurement,  production,  and  evaluation  of  JPAs  should  be  discussed  as 
a part  of  the  broader  technical  manual  problems.  This  provides  context 
within  the  requirements  for  the  logistic  support  elements  and  specifically, 
the  function  served  by  technical  manuals  for  maintenance  and  maintenance 
training.  If  we  ignore  these  continuing  problems,  the  potential  of  JPA 
effectiveness  may  never  be  realized.  After  all,  a JPA  is  a form  or  part 
of  a technical  manual;  it  is  procured  by  technical  manual  people;  it  is 
written  by  technical  writers  and  used  by  operators  and  maintainers. 

JPA  Production  and  Utility  Problems 

A survey  conducted  by  the  National  Security  Industrial  Association 
(NSIA)  and  reported  in  December  1974  identified  problems  of  importance  to 
the  utility  and  user  acceptance  of  technical  manuals.  These  problems  are 
closely  related  and  directly  affect  the  procurement,  production,  and  evalua- 
tion of  JPAs  for  training  operators  and  maintainers.  The  survey  subjects 
included : 

1.  Readability — Comprehensibility 

2.  Reading  Grade  Level 

3.  Presentation — Verbal,  Graphic,  Pictorial 

4.  User  Problems 

5.  User  Skill  Level 

6.  Critical  Job  Performance  Factors 

The  survey  was  initiated  to  develop  a data  base  for  recommendations  to 
the  Chief  of  Naval  Material.  Tne  recommendations  addressed  problems  of 
writing,  training,  operating,  and  maintenance  manuals  to  fit  the  average 
level  of  comprehension  of  Navy  personnel. 

Tlie  technical  manual  problems  identified  and  directly  related  to  the 
Navy  Personnel  and  Training  Utilization  of  Job  Performance  Aid  (JPA)  Tech- 
nology are  discussed  in  the  following  paragraphs. 

Organization  of  Technical  Manuals 

The  survey,  through  discuss^  ms  with  users  and  producers  of  technical 
manuals,  indicated  that  the  manual  organization  is  equal  in  importance  to 
factors  such  as  readability  and  comprehension.  The  following  excerpt  from 
the  introduction  to  NAVTRADEVCEN  69-C-0301-1  is  quoted:  "Technicians  ex- 

press a great  deal  of  frustration  with  ’just  trying  to  read  (maintenance 
manuals)  . . and  in  'trying  to  simply  find  out  what  is  in  them  . . 

The  significance  of  this  problem  to  the  procurement  and  production 
of  JPAs  is  that  the  equipment  maintenance  or  technical  manual  has  in  it 
sections  and  information  used  by  others  than  operation  and  maintenance 
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personnel.  For  example,  "General  Description"  is  a section  of  importance 
to  planners,  users,  and  others  requiring  information  not  covered  in  the 
Maintenance  Sections.  Therefore,  the  planning  and  procurement  of  tech- 
nical manuals  must  specify  what  sections  require  JPA  treatment,  what  sec- 
tions do  not,  where  the  overlap  is,  and  how  they  complement  each  other. 
Without  this  kind  of  systems  planning,  the  benefits  of  JPAs  are  lost  in 
the  maze  of  thousands  of  pages  of  manuals  describing  some  systems  (e.g., 
F-14  manuals  exceed  25,000  pages). 

Importance  of  Maintenance  Sections 


The  terms  "Technical  Manuals"  and  "Training,  Operating  and  Main- 
tenance Manuals"  are  too  general.  Various  studies  and  reports  bear  out 
the  conclusion  that  certain  sections  of  manuals  are  more  important  than 
others  in  the  maintenance  of  equipment.  These  are  "Action"  sections. 

They  include  check  out,  troubleshooting,  repair  and  replacement,  and 
other  job-oriented  maintenance  functions.  Sections  containing  theory, 
description,  parts  lists,  diagrams,  schematics,  and  drawings  are  refer- 
ence material. 

JPAs,  Job  Guides,  and  Logic  Tree  Troubleshooting  Aids  are  "Action" 
maintenance  aids.  They  should  be  identified  by  the  procurement  people  for 
JPA  treatment  during  the  initial  publications  planning  stage. 

Manuals  Contain  Too  Much  Data 


The  survey  indicated  that:  "Technical  manuals  contain  much  more 

data  than  is  required  by  the  maintainer  at  lower  levels  of  maintenance. 

The  number  of  pages  could  be  dramatically  reduced  if  the  needs  of  the 
maintainer  were  identified  and  coordinated  with  his  training." 

Manuals  are  identified  and  written  to  a specified  level  of  main- 
tenance. Today's  JPAs  have  been  largely  devoted  to  the  organizational  and 
intermediate  levels.  It  is  the  intermediate  level  that  contains  a variety 
of  material  that  is  not  action  type  or  performance  oriented.  This  material 
includes  diagrams,  tables,  and  drawings  not  greatly  improved  by  JPA  treat- 
ment . 


To  make  JPAs  where  they  are  not  applicable  or  useful  is  a waste  of 
time  and  money.  Unfortunately,  technical  manual  specifications  leave  to 
the  contractor  the  option  as  to  treatment  in  many  areas  without  direction 
from  the  government. 

Utility 

The  utility  of  a maintenance  manual,  or  the  quality  of  being  useful, 
depends  on  several  factors.  These  include  organization,  format,  technical 
accuracy,  the  use  of  illustrations,  and  many  others  such  as  media  and  tech- 
nique of  presentation.  The  utility  of  a maintenance  manual  also  depends 
on  weapon  system  logistic  support  interdependencies  at  each  level  of  main- 
tenance; that  is,  test  equipment  and  trained  personnel  (maintainers) . 
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It  is  in  the  area  of  utility  that  the  JPA  potential  seems  to  lie; 
more  so,  if  maintenance  training  can  be  conducted  using  the  actual  JPA 
that  will  be  in  the  technical  manual. 

Needs  of  the  Maintainer 


The  utility  of  the  maintenance  manual  is  directly  related  to  the 
background  of  the  maintainer.  It  is  a tool  he  may  or  may  not  use.  If  his 
background  includes  extensive  equipment-related  experience  in  isolating  a 
particular  fault,  he  needs  only  simple  test  equipment  or  limited  manual 
assistance.  He  is  guided  by  equipment  cues  and  his  background.  On  the 
other  hand,  the  novice  needs  all  the  help  he  can  get.  He  must  be  led  step- 
by-step  by  the  maintenance  procedures  in  checking  out  the  equipment  and 
in  the  use  of  the  test  equipment  to  isolate  the  faulty  component.  Ideally, 
the  maintenance  manual  would  complement  or  make  up  for  deficiencies  in  the 
maintainer's  education,  training,  and  experience. 

The  ideal  maintenance  manual  would  contain  only  action  information 
oriented  to  the  maintenance  task.  It  would  be  entirely  job  oriented  to 
serve  as  a maintenance  aid  to  the  maintainer.  All  other  data  would  be  re- 
organized and  included  in  reference  data  manuals  or  some  other  format  or 
media. 

Technical  Data  Development  Processes 

A recent  study  conducted  by  REM  Company  (Martin  & Johnson,  1975)  for 
the  Naval  Air  Development  Center  (NADC)  investigated  the  Technical  Manual 
and  Engineering  Data  Base  Interface.  This  report  was  reproduced  by  the 
Naval  Ships  Research  and  Development  Center,  Carderock,  Maryland  for  distri- 
bution. Pertinent  excerpts  or  comments  appropriate  to  the  procurement, 
production,  and  evaluation  of  JPAs  are  included  below. 

The  methods  the  defense  industry  typically  employs 
to  develop  technical  manuals  and  training  aids  for  main- 
tenance are  obtained  from  four  sources: 

1.  The  published  state-of-the-art 

2.  Historical  data  on  similar  systems 

3.  Engineering  data  on  the  system 

4.  Professional  expertise  of  technical  writer 

The  range  of  professionalism  in  technical  manual 
organizations  is  described.  Because  innovations  are 
deemed  to  have  minimal  effects  below  a certain  level  of 
professionalism,  this  task  report  uses  data  and  draws 
conclusions  based  on  the  methodology  of  companies  with 
technical  manual  departments  ranking  high  on  the  range 
of  professionalism  and  ranging  from  large  to  small  in 
size. 
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Both  the  substantial,  and  the  time-phase,  relations 
between  technical  manuals  and  training  aids  and  their 
primary  data  base  (engineering  data)  are  charted  and 
described.  It  is  noted  that  the  development  process 
for  both  technical  manuals  and  training  aids  begins 
with  the  generation  of  some  kind  of  an  intermediate 
data  base  from  the  primary  base  of  engineering  data 
and  the  professional  knowledge,  skill,  and  experience 
of  the  technical  manual  writer  and/or  training  aid  writer. 

The  vital  role  of  engineering  data  in  the  develop- 
ment of  technical  manuals  and  training  aids  is  described. 

It  is  demonstrated  that  engineering  drawings  are  a type, 
form,  or  precise  and  detailed  model  of  the  equipment. 

When  properly  made  and  available  in  a timely  manner, 
the  engineering  drawing  model  can  be  examined,  analyzed, 
manipulated,  and  exercised  to  develop  maintenance  infor- 
mation for  technical  manuals  and  training  aids  with  much 
of  the  work  completed  before  the  first  hardware  is  available. 

Engineering  drawing  schedules  coincide  closely  with 
hardware  manufacturing  schedules.  Seldom  are  technical 
manuals  and  training  schedules  contractually  reconciled  with 
each  other  and  with  the  schedules  for  engineering  and  manu- 
facturing. There  are  management  practices  and  techniques 
that  can  be  taken  to  minimize  these  difficulties  and  some 
companies  utilize  them  when  not  contractually  inhibited. 

Use  of  Technical  Manuals  in  Training 

The  DoD  customer  continues  to  raise  the  question,  "Why  can’t  tech- 
nical manuals  be  used  in  training?"  The  NADC  report  indicates  that  only  39 
percent  of  companies  surveyed  coordinate  training  manuals  with  training  re- 
quirements. The  reasons  cited  below  apply  to  problems  in  usage  of  the  JPAs 
in  maintenance  training: 

1.  Training  tends  to  overtraining  by  teaching  many  prerequisite 
courses  which  are  not  coordinated  with  technical  manuals. 

2.  Customer  schedule  requirements  for  equipment,  training,  and 
technical  manuals  are  not  structured  to  optimize  technical  manual  avail- 
ability based  on  engineering  data  release  and  on  validation;  and  training 
programs  are  not  scheduled  to  coincide  with  technical  manual  availability. 

3.  Customer  quality  requirements  are  not  the  same  for  technical 
manuals  and  training.  Training  can  be,  and  is,  conducted  with  data  and 
material  that  lags  considerably  behind  the  hardware  configuration  to  be 
delivered.  On  the  other  hand,  technical  manuals  must  be  accurate  and  con- 
figuration accountable.  Because  of  the  dialog  in  face-to-face  confron- 
tation between  instructor  and  pupil,  many  shortcomings  in  training  material 
can  be  easily  compensated;  something  that  is  not  possible  with  technical 
manuals.  This  quality  gap  tends  always  to  keep  technical  manuals  in  lagging 
phase  with  respect  to  training. 
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4.  Technical  manuals  emphasize  theory  too  much  and  diagnostics  and 
repair  too  little. 

CDRL  Interdependencies 

DoD  requirements  are  specified  in  a Contract  Data  Requirements 
List  (DD-1423).  These  data  items  are  shown  in  Figure  10.  Common  to  every 
element  is  engineering  data. 
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In  dealing  with  the  timeliness  of  data,  suggestions  are  made  con- 
cerning those  actions  that  both  the  military  customer  and  the  contractor 
can  make  to  assure  timeliness  of  data.  Contributing  to  timeliness  is 
common  usage  of  data  items  by  functional  organizations  of  the  customer 
and  the  contractor,  especially  the  latter.  Where  common  data  item  usage 
is  not  imposed,  each  functional  organization  of  the  contractor  will  respond 
to  schedule  pressures  by  developing  (insofar  as  possible)  its  own  band 
of  engineering  data.  This  practice  can  be  costly  and  sometimes  does  not 
permit  even  a low  quality  product  or  service  to  be  delivered  on  time 
because  final  delivery  is  contingent  on  release  of  the  authorized  engineer- 
ing data.  Common  usage  of  data  items  (which  requires  customer  acquiescence) 
reduces  data  volume  and  increases  the  incentive  for  its  timely  development. 

It  is  suggested  that  technical  manual  management  milestone  their 
engineering  data  requirements  in  close  time-phase  relationship  with  the 
realistic  release  schedule  for  engineering  data. 

To  complicate  matters,  it  is  not  only  the  release  dates  of  engineer- 
ing drawings  that  affect  schedules.  The  availability  of  design  engineers, 
both  to  supplement  engineering  drawing  information  and  to  review  manuscript, 
can  have  an  equally  strong  impact.  One  of  the  strongest  influences  on 
both  schedule  and  quality  is  the  time  of  availability  and  the  access  times 
to  the  weapon  system  hardware  and  the  support  equipment  to  be  used  in  its 
maintenance.  Poor  scheduling  or  schedule  slippages  can  delay  manuscript 
writing  and  manuscript  validation. 

Data  Sources  and  Uses 


The  NADC  report  (Martin  & Johnson,  1975)  lists  the  following  per- 
tinent data  for  technical  writing:  (1)  laboratory  notebooks,  (2)  technical 

memoranda,  (3)  test  data,  (4)  engineering  drawings,  (5)  equipment,  and 
(6)  specifications  and  standards. 

Technical  writers  would  be  lost  without  access  to  engineering  and 
manufacturing  drawings.  The  role  of  engineering  data  is  to  serve  the  tech- 
nical writer  and  training  instructor  as  the  only  available  model  of  the 
hardware  until  It  has  been  manufactured.  Ideally,  technical  manuals  and 
training  materials  are  best  developed  from  maintenance  task  analyses  con- 
ducted on  the  hardware  to  be  maintained. 

There  is  the  universal  need  to  have  available,  trained  maintenance 
technicians,  equipped  with  maintenance  instructions  with  the  delivery  of  the 
first  equipment  to  the  customer.  This  is  as  true  of  washing  machines  and 
automobiles  as  it  is  of  weapon  systems.  The  satisfaction  of  this  need  has 
entailed  the  development  of  special  engineering  disciplines,  commonly  called 
maintenance  engineering  and  maintenance  task  analysis,  that  are  applied  to 
the  model  represented  by  the  engineering  data. 

The  role  of  engineering  data  in  this  process  is  often  misunderstood. 
Many  laymen  think  that  engineering  drawings  contain  all  the  knowledge  to 
be  imparted  in  training  and  technical  manuals  and  that  it  only  needs  to 
be  extracted  and  reformatted  for  training  and  technical  manuals.  This 
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belief  is  wholly  incorrect.  Engineering  drawings  contain  a wealth  of  infor- 
mation about  the  equipment  that  is  essential  to  the  content  of  training 
courses  and  technical  manuals.  But  engineering  drawings  do  not  explain 
how  anything  works;  they  do  not  specify  what  can  fail  or  how  to  trouble- 
shoot failures;  and  they  do  not  contain  procedures  for  maintenance. 

In  short,  engineering  drawings  portray  what  is  to  be  manufactured 
but  they  do  not,  as  a rule,  tell  how  to  manufacture  or  how  to  maintain. 
Contractors  have  process  specifications  and  other  documented  manufacturing 
procedures  usually  referenced  on  engineering  drawings  or  else  required  by 
operations  manuals.  These  documents,  when  supplemented  with  Manufacturing 
Information  developed  for  each  set  of  engineering  drawings,  make  them  cap- 
able to  direct  the  manufacturing  process.  No  such  data  bank  exists  for 
training  materials  and  technical  manuals.  Thus,  all  of  this  kind  of  in- 
formation must  be  developed  based  on  analyses  of  the  engineering  drawings. 

Production  and  Evaluation  of  JPAs 


The  processes  involved  in  producing  technical  manuals  or  JPAs  are  ex- 
tremely complex  and  human  error  is  possible  at  each  step.  There  are,  for 
example,  over  50  major  work  steps  between  the  time  the  contractor  receives 
the  work  statement  and  authorization  from  ILS  manager  and  the  final  negatives 
of  a JPA  are  shipped  to  the  printer.  A flow  chart  of  a typical  development 
process  for  technical  manuals  and  training  materials  is  contained  in  the 
NADC  report. 

The  validation  and  verification  of  technical  manuals  and  JPAs  must  be 
planned  and  implemented  from  the  outset  of  planning.  Validation,  in  the 
strict  meaning  of  the  word,  is  the  contractor's  evaluation  of  his  written 
maintenance  procedures  by  performing  them  on  the  actual  hardware.  Simula- 
tion and  desk-top  substitutes  for  100  percent  hands-on  validation  have  a 
very  low  yield  (if  any  at  all)  in  benefits.  Not  only  does  the  hardware 
liave  to  be  available  for  validation  but  it  must  also  have  the  deliverable 
configuration  and  it  must  be  completely  operable.  In  addition,  and  of  equal 
importance,  all  GFE,  all  support  and  test  equipment,  tools,  and  spare  parts 
must  be  available.  Hardware  availability  for  validation  must  occur  about 
two-thirds  of  the  way  through  a program  for  concurrent  delivery  of  adequate 
technical  manuals  and  for  their  effective  use  as  a training  aid  for  the 
first  class  trained  by  the  contractor. 

It  is  possible,  of  course,  to  produce  some  kind  of  technical  manual 
from  the  engineering  data  and  use  it  without  validation  and  verification. 

This  has  frequently  been  done  and  the  practice  is  one  of  the  major  causes 
for  generally  poor  technical  manual  quality.  But  the  ease  with  which  this 
approach  can  be  taken  is  probably  responsible  for  the  failure  of  government 
to  schedule  realistically. 

Obviously,  if  a good  validation  Is  to  be  performed,  planning  and  the 
logistical  work  to  make  hardware  available  and  accessible  must  begin  early 
and  continue  unabated  until  validation  is  accomplished.  Of  equal  importance 
is  the  need  to  have  access  to  the  equipment  in  reasonable  states  of  assembly 
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for  100  percent  hands-on  task  analysis  (performance  of  and  critical  be- 
havioral evaluation  of  each  maintenance  procedure)  in  the  writing  phase. 

If  this  cannot  be  done,  there  are  only  two  recourses  known  to  the  industry 
at  this  time.  One  is  to  delay  all  procedural  writing  and  do  it  at  valida- 
tion. The  other  is  to  write  to  the  best  judgment  of  the  technical  writer 
and  rewrite  (usually  70%)  at  validation.  Neither  approach  is  really  accep- 
table but  the  latter  must  be  firmly  avoided  to  avoid  major  budget  overruns. 
Neither  approach  is  calculated  to  produce  a good  technical  manual  for  the 
timely  use  of  training. 


USER  PROBLEMS  IN  JPA  UTILIZATION 


Reid  P.  Joyce 

Applied  Science  Associates,  Inc. 


Introduction 

Almost  exactly  10  years  ago,  the  Air  Force  hosted  a performance-aids 
symposium  much  like  the  present  one.  Most  of  the  attendees  were  familiar 
with  each  other's  work,  so  the  assigned  topics  were  dealt  with  quite  rapidly, 
and  the  bulk  of  the  discussion  focused  on  a common  concern:  frustration 

with  the  difficulties  of  getting  our  research  findings  implemented.  There 
followed  a period  of  several  years  of  zealous,  hard-sell  activity  during 
which  the  JPA  community  made  quite  a variety  of  enemies  among  military  pub- 
lications and  logistic-support  organizations,  and  few  of  our  JPA  concepts 
were  adopted.  Finally,  within  the  last  4 or  5 years,  accompanied  by  little 
or  no  fanfare,  there  has  been  a quiet  infusion  of  JPA-like  features  into 
the  maintenance  documentation  systems  of  most  of  the  services.  Although 
it  is  gratifying  to  see  some  of  our  ideas  finally  getting  into  the  field, 
the  satisfaction  is  a bit  hollow:  some  technical  manuals  look  more  like 
JPAs  now,  but  relatively  few  of  the  potential  benefits  of  JPAs  are  actually 
being  realized. 

One  of  the  areas  extensively  researched  over  the  years  has  been  that 
of  "user  acceptance."  We  explored  many  features  of  tech  data  and  its  pre- 
sentation that  affect  the  extent  to  which  a user  "likes"  his  performance 
aids.  Audiovisual  presentations  have  been  compared  with  books.  Books  have 
been  varied  in  size,  shape,  color,  contrast,  and  format.  Content  has  been 
varied  from  fully  procedural  to  nonprocedural.  We  discovered  that  people 
who  have  to  move  around  a lot  like  portable  aids,  and  that  people  who  work 
in  dim,  dark  environments  need  bigger  printing.  And  we  discovered  that 
people  with  hardly  any  training  at  all  can  solve  very  complicated  problems, 
and  enjoy  doing  it,  if  they  are  given  enough  information  in  their  performance 
aids.  We  established  all  of  these  things  quite  conclusively  through  a long 
series  of  studies  and  experiments  of  which  we  can  be  justifiably  proud.  I 
believe  that  today's  JPA  technology  base  provides  us  with  a good  collection 
of  solutions  to  the  problems  of  building  performance  aids  that  the  user 
can  use,  and  that  he  will  want  to  use. 

So  what  is  the  problem?  Why  isn't  everyone  using  JPAs?  It's  because 
the  man  who  uses  a JPA  on  the  job  is  only  a small  part  of  a much  larger 
system  of  support  people  and  organizations,  and  we  haven't  yet  shown  the 
world  just  where  JPAs  fit  in  that  system.  Even  worse,  some  of  our  experi- 
ments have  left  some  people  with  the  feeling  that  JPAs  are  more  of  a monkey 
wrench  than  a panacea.  The  subjects  of  many  of  our  studies  have  performed 
admirably  in  the  lab  and  the  field;  but,  after  we've  made  our  point  and 
gone  away,  it's  become  clear  to  our  hosts  that  they  are  stuck  with  some 
rather  peculiar  people,  for  whom  there  is  no  place  in  the  system. 

Clearly,  we  have  reached  a point  where  we  have  to  shift  our  focus  a 
bit  when  we  consider  "user  problems,"  and  the  list  of  topics  for  the  pre- 
sent symposium  reflects  that  shift.  Integration  is  now  the  name  of  the 


game,  and  the  "user"  who  needs  our  attention  most  is  the  system  itself, 
not  just  the  guy  with  the  book.  If  we  want  to  see  JPA  concepts  pay  off, 
we  have  to  conceptualize  a system  in  which  JPAs  and  their  users  have  a 
clearly  defined  role,  one  that  is  compatible  with  the  rest  of  the  system. 
The  problem  areas  that  make  up  the  rest  of  this  paper  consequently  tend  to 
address  the  needs  of  the  using  system  more  than  the  using  individual. 

Program  Management  Level  Problems 


Program  managers  have  had  the  authority  to  select,  during  system  develop- 
ment, from  a limited  range  of  technical  documentation  alternatives.  Those 
who  stick  strictly  with  tradition  are  seldom  criticized;  those  who  innovate 
have  sometimes  been  burned.  Even  a manager  who  is  knowledgeable  about  both 
traditional  and  innovative  performance  aids  still  has  little  in  the  way  of 
hard  data  and  clear-cut  guidance  upon  which  to  base  a comfortable  decision 
about  the  best  mix  of  tech  data  for  his  system.  He  has  even  less  informa- 
tion available  to  him,  and  less  real  authority,  to  specify  the  best  person- 
nel system — especially  training  and  career  patterns — to  complement  the  per- 
formance aids  he  wants. 

Specifications  are  Incomplete 

Although  some  of  the  services  have  incorporated  some  JPA  features 
into  tech  data  specifications,  we  have  yet  to  construct  a spec  that  goes 
beyond  the  JPAs  to  show  the  manager  a good  "impedance  match"  between  the 
JPAs  and  the  rest  of  the  system.  Some  of  the  draft  specs  include  speci- 
fication of  certain  parts  of  the  JPA  development  process . Many  of  us  be- 
lieve that  the  procurement  activity  must  monitor — and  have  some  authority 
over — the  development  process  In  order  to  assure  completeness  and  accuracy 
in  JPAs,  but  this  tends  to  spook  some  managers,  who  aren’t  used  to  meddling 
much  in  the  tech  data  until  pretty  far  downstream. 

The  perceptive  manager  contemplating  an  existing  JPA  spec  will 
recognize  that: 

1.  If  he  implements  JPAs  without  finding  the  proper  niche  for 
them,  he  won't  get  much  extra  benefit  for  the  money  he  spends. 

2.  The  spec  provides  little  or  no  guidance  to  help  him  decide 
what  kind  of  niche  to  make  or  how  to  exercise  the  necessary  quality 
control . 

Procurement  and  Updating  Costs  are  High 

A manager  who  has  been  around  for  a while  has  probably  heard  some 
horror  stories  about  huge  costs  associated  with  JPAs,  especially  those  that 
include  extensively  proceduralized  troubleshooting.  The  fact  that  we  haven’t 
yet  defined  the  optimum  system  role  for  such  aids  makes  their  admittedly  high 
cost  understandably  hard  to  swallow.  Furthermore,  we  haven't  gone  far 
enough  in  describing  updating  procedures.  Logistics  people,  who  have  a 
hard  enough  time  keeping  conventional  publications  current,  recognize 
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the  potentially  massive  impact  that  even  relatively  small  hardware  changes 
can  have  on  a collection  of  JPAs.  Again,  recognizing  the  potential  benefits 
of  JPAs  used  on  the  job  just  isn't  enough  to  prevent  managers  and  logistics 
people  from  being  scared  away  by  vague  and  mostly  incomplete  guidance  in 
acquisition  and  control  of  the  JPAs  over  the  life  of  the  system. 

Tradition  is  Hard  to  Beat 


The  conventional  technical  manual  in  most  of  the  military  services 
carries  with  it  the  weight  of  tradition  and  the  inertia  of  long-term  fami- 
liarity within  large  logistic-support  organizations.  Even  when  it  isn't 
rationally-based,  the  use  of  tradition  as  an  argument  for  the  status  quo 
will  be  much  harder  to  counter  until  we  can  show  the  world  a complete  new 
system  that  describes  all  of  the  differences  from  the  traditional  one.  As 
long  as  we  persist  in  providing  a kit  with  some  of  the  essential  pieces 
missing,  we'll  continue  to  have  trouble  even  getting  the  traditionalists' 
attention. 

What  About  Backup? 

To  the  extent  that  JPAs  are  either  incomplete  or  inaccurate,  the 
system  must  have  alternate  means  to  heal  itself.  An  unfortunate  flaw  in 
the  early  JPA  sales  pitch  was  that  it  pretty  much  ignored  this  question. 

JPA  was  sold  as  a panacea  in  fully-proceduralized  form,  but  thoughtful 
managers  recognized  that  if  fully  proceduralized  aids  were  the  only  system 
documentation  available,  if  they  failed  to  solve  a problem  (regardless  of 
the  reason),  and  if  the  technicians  were  wholly  dependent  on  the  JPAs,  un- 
able to  apply  some  "creative"  troubleshooting,  the  whole  system  would  shortly 
collapse.  Studies  conducted  to  date  reinforce  the  feasibility  of  integra- 
tion of  the  JPA  approach  into  the  documentation  system,  but  not  replacement 
of  the  entire  existing  system  with  JPAs — an  intent  that  we  allowed  many 
people  to  erroneously  infer  from  our  early  enthusiasm.  Other  papers  pre- 
sented at  this  symposium  have  described  in  more  detail  some  of  the  currently- 
proposed  solutions  to  this  problem,  including  restructuring  career  fields 
and  personnel-advancement  schemes.  The  ultimate  solution  will  probably 
include  JPAs  used  by  inexpensively-trained  early-enlistment  people,  and 
more  conventional  aids  used  by  more  experienced  people  who  have  received 
supplemental  training  along  the  way,  or  who  go  to  another  completely  dif- 
ferent resident  school,  perhaps  at  the  beginning  of  their  next  enlistment. 

A fairly  large  group  of  the  former  kind,  and  a smaller  (through  attrition) 
group  of  the  latter  kind  could  probably  cover  all  bases,  fairly  economically. 

What  About  Nonmaintenance  Users? 

Another  reason  why  program  managers  have  been  frightened  away  from 
JPAs  is  their  recognition  of  the  fact  that  some  JPA  specs  completely  ignore 
nonmaintenance  users  of  the  documentation  system.  Again,  we  failed  to  make 
it  clear  that  most  of  our  innovative  maintenance  aids  were  intended  to  re- 
place or  supplement  only  those  portions  of  conventional  manuals  that  tech- 
nicians use,  rather  than  to  supplant  the  entire  system.  Supply  and  trans- 
portation people,  operators,  design  and  service  engineers,  and  curriculum 
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planners,  to  name  a few,  all  have  legitimate  interests  in  information  about 
the  system  that  might  not  be  included  in  JPAs  for  maintenance,  yet  some  of 
the  JPA  specs  have  provided  little  or  no  guidance  to  the  manager  in  choos- 
ing the  right  mix  of  conventional  and  JPA-type  material. 

Field  User  Level  Problems 

As  mentioned  earlier,  most  of  the  problems  we  have  seen  in  recent  years 
with  rejection  of  JPAs  by  users  seem  to  derive  more  from  organizational  and 
social  features  than  from  unsolved  technical  difficulties  with  the  aids  them- 
selves. If  people  lose  confidence  in  their  performance  aids,  or  if  they 
feel  that  their  jobs  are  somehow  demeaning,  then  they're  in  trouble  no 
matter  how  good  the  aids  actually  are. 

JPAs  Need  a Proper  Introduction 

We  have  probably  all  seen  examples  of  a good  idea  being  defeated 
even  before  it  gets  started.  The  two  most  common  pieces  of  scuttlebutt 
that  are  likely  to  precede  a JPA- implementation  exercise  are: 

1.  They're  so  full  of  errors  that  you  can't  use  them — even 
worse  than  conventional  manuals. 

2.  They're  designed  for  use  by  dummies,  so  if  you  get  them,  it 
means  they  think  you're  not  smart  enough  to  figure  it  out  for  yourself. 

Little  wonder  that  people  are  at  least  suspicious,  and  that  they  tend  to 
nit-pick  a bit.  The  only  countermeasure  we  can  apply  is  a carefully  pre- 
sented introduction  that  explains  that  our  aids  don't  have  the  errors 
that  earlier  experimental  models  had,  and  that  shows  everyone  how  the 
JPA-user  fits  into  the  organization  and  what  his  future  options  are.  All 
this  presupposes,  of  course,  that  the  organization  really  does  have  a place 
for  hi^,  and  tluit  his  career  field  isn’t  planned  to  be  a uead-end  slot  for 
dummies.  People  who  are  exposed  to  JPAs  right  after  tech  school,  as 
Mr.  Klesch  will  describe  later,  are  painfully  aware  of  how  little  they 
have  really  been  taught  to  do,  and  they  are  extremely  receptive  to  pro- 
cedural aids.  They  see  the  aids  as  giving  them  the  ability  to  do  produc- 
tive work  right  away,  rather  than  depending  on  an  extensive  period  of 
hand-holding  by  their  experienced  co-workers. 

It  may  be  a fairly  fine  distinction,  but  we've  discovered  that 
even  experienced  people  tend  to  accept  procedural  aids  quite  readily  if 
they  are  billed  as  something  to  make  the  job  easier — not  simpler.  People 
in  supervisory  positions  get  uneasy  when  something  comes  along  that  tends 
to  minimize  the  complexity  of  their  operations.  Shop  chiefs  pride  them- 
selves in  being  able  to  say  that  they  possess  the  very  finest  technicians, 
and  only  through  superhuman  effort  are  they  able  to  keep  their  incredibly 
complex  gear  on  line.  As  long  as  that  makes  a good  story,  we're  better 
off  to  appear  to  be  fighting  the  battle  against  complexity,  not  reducing 
troubleshooting  to  child's  play. 
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Users  Must  be  Trained  in  JPA  Use 


This  is  really  no  different  from  introducing  people  to  JPAs,  but  we 
have  occasionally  stopped  short.  People  need  not  only  to  hear  about  what 
the  aids  do,  but  they  must  also  practice  using  them  properly.  This  prac- 
tice should  be  followed  up  in  the  field  to  assure  that  later  questions  get 
answered,  and  that  the  aids  don't  fall  into  disuse  because  the  users 
never  quite  caught  on  to  all  the  tricks  before  the  salesman  left.  I’ve 
seen  SIMM-type  documentation  lying  around  completely  unused,  simply  because 
the  technicians  admit  that  they  never  really  learned  how  to  use  them.  These 
aids  are  pretty  to  look  at,  and  they  impress  the  visitors,  but  the  users 
just  don't  feel  that  they  have  the  time  to  teach  themselves  how  to  use  them. 

Similarly,  even  modestly-experienced  technicians  have  a tendency 
not  to  believe  that  a f ully-proceduralized  troubleshooting  book  is,  as  its 
name  imples,  a thing  that  must  be  followed  to  the  letter.  They  tend  to 
omit  steps  that  appear  to  be  unnecessary  and  to  try  to  take  other  short- 
cuts. When  the  procedures  subsequently  bomb,  the  technicians  conclude  that 
it's  the  procedures'  fault,  and  they  discontinue  using  them.  It  is  a rela- 
tively simple  task  to  demonstrate  the  devastating  effect  of  trying  to  ad 
lib  with  such  procedures,  but  we  often  fail  to  do  so. 

Supervisor  and  Peer  Acceptance  of  JPA  Users 

We've  gotten  ourselves  into  this  bag  repeatedly  with  some  of  our 
field  studies,  and  the  problem  will  not  get  better  until  we  get  the  rest 
of  the  system  revised.  We  have  had  a tendency  to  demonstrate  some  of  our 
principles  by  dragging  people  out  of  boot  camp,  showing  them  how  to  solder 
without  hurting  themselves,  giving  them  a package  of  experimental  JPAs,  and 
measuring  their  performance.  In  at  least  a couple  of  instances,  these  people 
have  somehow  managed  to  stay  in  the  system,  at  least  for  a while,  after  the 
study  was  over.  It  doesn't  take  long  for  supervisors  and  peers  to  peg  these 
folks  as  the  "dummies,"  because  they  just  don't  talk  like  "real"  technicians, 
even  though  they  may  perform  every  bit  as  well  with  their  JPAs.  The  kicker 
comes  when  it's  time  to  qualify  for  the  next  skill  level.  Not  only  do  they 
have  lower  supervisor  ratings  (because  they  talk  funny),  but  they  find  that 
they  can't  possibly  compete  with  their  conventionally  trained  peers.  Most 
career-progression  ladders  are  based  on  advancement  criteria  that  discriminate 
against  the  JPA  user  by  testing  for  "knowledge"  that  has  not  been  conveyed 
by  his  schooling,  because  he  doesn't  need  the  knowledge  to  do  the  job  with 
JPAs. 


Ex-students  who  could  see  this  problem  coming  have  actually  gotten 
to  the  point  of  conferring  with  their  congressmen  about  the  effect  that 
such  an  experiment  has  had  on  their  careers. 

Generalizabillty  of  Skills 


Certainly,  people  learn  as  they  use  JPAs,  particularly  from  non- 
troubleshooting procedural  material.  Some  tasks  can  be  learned  fairly 
quickly,  and  some  JPA  formats  provide  a dual-level  presentation  that  allows 
the  inexperienced  man  to  follow  every  step  but  cues  the  experienced  man 


at  a somewhat  higher  level.  Because  the  principles  of  mechanical  and 
electrical  assembly  and  repair  tend  to  be  reflected  in  a fairly  consistent 
way  within  most  systems,  some  JPA  users  find  that  they  can  figure  out  how 
to  do  a variety  of  things  that  are  similar  to  ones  for  which  they've  used 
JPA  procedures.  In  this  sense,  then,  some  of  the  specific  operations  that 
a user  practices  as  he  follows  a procedure  can  generalize  to  other  tasks 
for  which  he  does  not  have — or  doesn't  choose  to  use — a written  procedure. 

Although  some  kinds  of  semi-procedural  or  nonprocedural  trouble- 
shooting aids  can  foster  the  development  of  troubleshooting  skills  (an 
essentially  cognitive  activity),  fully  proceduralized  troubleshooting  does 
not.  Troubleshooting  procedures  typically  provide  neither  a map  through 
the  data  flow  nor  reasons  for  the  checks.  The  troubleshooter  who  is  wholly 
dependent  upon  the  procedure  is  unlikely  to  learn  anything  at  all  about  the 
principles  behind  the  procedural  sequence.  We  know  how  to  build  both  kinds 
of  aids,  so  the  choice  must  be  made  on  the  grounds  of  how  much  we  wish  to 
spend  on  training,  and  what  kind  of  personnel  system  and  career  advancement 
structure  we  wish  to  have. 


NEW  DIRECTIONS  FOR  INFORMATION  TRANSFER 
RESEARCH  IN  MAINTENANCE  JOBS 


Edgar  L.  Shriver 
Kinton,  Incorporated 

Introduction 

Maintenance  personnel  need  information  to  get  their  jobs  done.  Tradi- 
tionally, information  has  been  subdivided  into  compartments  called  train- 
ing and  technical  manuals.  The  traditional  philosophy  is  that  personnel 
should  be  trained  in  the  fundamentals  and  that  manuals  should  contain  the 


system  specifics.  It  is  assumed  that  personnel  will  then  apply  their  fun- 
damental knowledge  and  skills  to  specific  technical  information  and  deduce 
what  actions  should  be  taken  in  each  situation  on  the  job. 

Twenty  years  of  research,  development,  and  some  implementation  have 
eroded  the  distinction  between  these  compartments.  Today,  there  is  a new 
philosophy  for  producing  documentation.  It  holds  that  training  documents 
should  tell  maintenance  personnel  what  actions  to  take  on  specific  equip- 
ment under  various  conditions.  This  new  technical  documentation  takes  over 
a large  percentage  of  what  training  and  training  devices  were  expected  to 
mediate,  and  the  test  standard  and  conditions  also  are  built  into  the  book. 

This  paper  will  review  what  has  been  found  to  be  effective,  what  the 
life  cycle  cost  implications  are,  what  remains  to  be  determined,  and  where 
the  field  must  go  to  achieve  maximum  payoff  in  the  operational  world. 

First,  let's  look  at  costs.  One-third  of  all  personnel  in  the  Armed 
Services  are  in  full-time  maintenance  jobs.  This  does  not  include  equip- 
ment operators  who  also  do  some  maintenance.  The  cost  of  an  enlisted  per- 
son varies  with  rank  and  with  support  costs  (e.g.,  food,  housing,  health 
care,  and  retirement).  We  use  a figure  of  $30,000  per  year  as  an  estimate, 
though  $25,000  or  $35,000  or  even  $40,000  are  also  used  as  estimates  by 
others.  There  are  about  two  and  one-half  million  enlisted  personnel  in  the 
services.  With  one-third  of  them  in  maintenance  jobs  at  $30,000  per  year, 
this  amounts  to  more  than  18  billion  dollars  per  year  in  maintenance  per- 
sonnel costs.  This  is  about  six  billion  dollars  for  each  of  the  three 
services.  The  cost  of  all  technical  documentation  operations  is  no  more 
than  3 percent  of  this  cost  in  each  service  per  year. 

About  one-fourth  of  the  maintenance  personnel  is  new  to  the  service 
each  year.  An  average  of  20  weeks  is  spent  in  nonproductive  maintenance 
training  hefore  these  personnel  do  any  productive  work  on  the  job.  For 
some  6 months  on  the  job,  their  productivity  is  low.  The  student/instruc- 
tor ratios  for  training  are  on  the  order  of  1-to-l  or  3-to-l.  There  is 
steady  Congressional  pressure  to  reduce  this  ratio.  The  cost  of  maintenance 
training  for  first  enlistment  personnel  is  on  the  order  of  a half  billion 
dollars  per  year  per  service.  Another  billion  per  year  per  service  is  at- 
tributable to  the  nonproductive  time  of  personnel  in  training  and  the  reduced 
productivity  of  personnel  during  their  first  6 months  on  the  job  (assuming 
an  average  15%  reduction  in  productivity  during  the  6 months).  These  are 
the  raw  dollar  figures. 
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Our  next  question  is  where  cost  can  be  reduced  by  changing  the  infor- 
mation transfer  methodology  available  today.  Studies  over  the  past  20  years 
show  reductions  in  training  on  the  order  of  50  to  90  percent.  Increases 
in  job  productivity  indicated  in  these  studies  have  been  as  high  as  50  and 
60  percent.  Many  different  conditions  combine  to  produce  this  range  of 
values  (e.g.,  type  of  task  trained,  original  length  of  training,  etc.). 

But  these  are  the  ball  park  reductions  that  can  be  expected  from  new  infor- 
mation transfer  methods  as  projected  from  research  studies  over  the  past  20 
years  (Shriver  & Hart,  1975).  Savings  on  the  order  of  one  to  two  billion 
per  year  per  service  are  implied  by  these  reductions. 

That  sounds  good  for  reductions.  What  are  the  increased  costs?  The 
increased  costs  are  in  producing  the  manuals  that  tell  the  maintenance 
, personnel  what  to  do  on  the  job.  The  contracts  for  such  new  manuals  in 

the  Army  MIL-M-632XX  are  running  about  as  much  as  for  old  style  manuals  or 
somewhat  more.  The  costs  of  JPA  manuals  in  the  Air  Force  are  similarly 
20  to  50  percent  higher  than  conventional  manuals.  These  increased  costs 
represent  no  more  than  a million  dollars  per  major  system.  This  is  a neg- 
ligible amount  for  a new  system  when  the  reductions  are  so  much  greater. 

The  cost  of  reworking  all  existing  manuals  is  higher.  It  would  not  be 
prohibitive  if  the  cost  savings  were  ensured,  but  more  data  are  needed  to 
reduce  risk. 

The  Study  Conditions  and  State-of-the-Art  Today 

Reasons  for  Reduced  Training  and  Increased  Productivity 

A recent  study  of  20  years  of  research  identified  five  fundamental 
elements  which  account  for  the  effectiveness  of  new  concepts  in  manuals 
and  training.  They  are: 

1.  Equipment  Analysis.  This  is  an  indentured  list  of  all  parts 
in  the  system  as  entries  in  a matrix.  The  matrix  has  column  headings  re- 
ferring to  about  ten  kinds  of  maintenance.  The  matrix  cell  entries  contain 
a code  indicating  the  level  or  echelon  of  maintenance  that  provides  the 
maintenance  action.  • iis  is  called  the  Task  Identification  Matrix  (TIM) 
in  Air  Force  Job  Performance  Aid  (JPA)  specifications,  and  the  Maintenance 
Allocation  Chart  (MAC)  in  the  Army.  The  Navy  does  not  have  a direct  analogue, 
but  the  Maintenance  Dependency  Chart  (MDC)  for  troubleshooting  in  SIMMS  and 
FOMM  specifications  has  certain  similarities. 

2.  Functional  Analysis.  This  is  known  by  many  terms:  cause/effect 

analysis,  failure  mode  analysis,  logic  trees,  fault  isolation  charts,  etc. 
There  are  similarities  and  differences  among  all  the  specific  techniques, 
but  they  all  have  the  following  items  in  common:  an  analysis,  a logical 

relating  of  part  failures  to  indications  of  failure,  and  a guide  for  user 
maintenance  actions  on  the  basis  of  indications.  They  all  apply  to  trouble- 
shooting maintenance  only  and  represent  a way  of  codifying  a process  that 
was  traditionally  regarded  as  a problem-solving  situation. 

3.  Task  Analysis.  This  is  describing  in  detail  the  steps  to  be 
taken  in  maintenance  procedures  when  each  step  invariably  follows  another 
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to  accomplish  a task.  It  includes  a description  of  the  condition  under 
which  the  tasks  are  performed  and  the  needed  tools.  It  is  used  for  all 
nontroubleshooting  tasks  and  the  procedural  aspects  of  troubleshooting 
tasks . 


4.  Behavioral  Task  Analysis.  This  is  a refinement  of  Task  Analysis 
in  which  the  cues  are  specifically  identified  for  the  more  detailed  actions 
identified  in  the  grosser  task  analysis.  It  includes  use  of  graphics  for 
showing  cues. 

5.  Intelligibility  Standards.  This  provides  a process  for  making 
the  graphic  and  written  instructions  intelligible  to  personnel  with  grade 
school  reading  ability.  It  includes  a verb  list  of  single  syllable  verbs 
(actions),  which  are  allowed  in  instructions.  It  specifies  how  graphic 
cues  take  the  place  of  nouns  and  how  words  relate  to  pictures.  The  sen- 
tence length  is  also  restricted.  It  is  a process  for  forcing  intelligi- 
bility rather  than  for  measuring  intelligibility  after  a product  is  pro- 
duced . 


It  was  found  that  the  increased  effectiveness  of  personnel  and  the 
reductions  in  training  time  in  25  studies  could  be  traced  to  one  of  these 
elements  or  some  combination  of  them.  It  was  also  seen  that  all  the  tech- 
niques could  be  applied  in  each  application;  that  is,  they  do  not  compete 
with  each  other.  The  exception  is  functional  analysis.  It  has  application 
only  in  troubleshooting  situations.  We  might  look  at  applications  of  these 
analytic  techniques  in  each  of  the  services  for  some  illustrative  examples. 

The  Army  was  first  in  1956  with  FORECAST.  Research  prior  to  that 
time  was  based  on  training  only.  FORECAST  specifically  dealt  with  manuals 
and  training  plus  training  devices  and  performance  testing.  The  Army  used 
the  Maintenance  Allocation  Chart  (MAC)  at  that  time  so  the  research  never 
dealt  with  the  first  fundamental  element.  FORECAST  dealt  only  with  elec- 
tronic troubleshooting.  It  used  functional  analysis,  task  analysis,  and 
a rudimentary  form  of  behavioral  task  analysis  for  its  effects.  It  also 
used  short  words  but  not  according  to  a prescribed  list. 

SIMMS  with  its  Maintenance  Dependency  Chart  (MDC)  came  next.  It 
started  in  the  Coast  Cuard  and  then  was  adopted  by  the  Navy.  The  MDC  used 
functional  analysis  and  a form  of  equipment  analysis  for  its  effects. 

Words  tended  to  be  short,  but  special  symbology  was  used  to  communicate 
logical  relationships.  In  its  original  form,  it  was  designed  for  elec- 
tronic troubleshooting  but  was  applied  to  mechanical  systems  as  well. 

PIMO  followed  very  closely  in  the  Air  Force  about  1960.  It  used 
behavioral  task  analysis  and  intelligibility  standards  for  its  effect. 

It  was  designed  to  show  locations  and  appearances  of  mechanical  system 
parts  in  graphic  terms  with  short  words  and  sentences.  The  prescribed  word 
list  was  originated  in  this  concept.  PIMO  was  the  first  new  concept  to 
focus  on  mechanical  maintenance.  For  troubleshooting  tasks,  the  MDC  was 
lifted  from  SIMM  and  used  by  PIMO.  PIMO  tests  included  a lot  of  opinion  data 
from  users.  The  key  finding  was  that  new  users  liked  the  information  while 
experienced  personnel  disliked  the  detailed  instruction. 
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The  JPA  concept  is  an  amalgam  of  PIMO  and  fully  proceduralized 


troubleshooting  procedures.  It  used  the  fundamental  elements  of  equip- 
ment analysis,  functional  analysis,  task  analysis,  and  intelligibility 
standards  for  its  effects.  The  term  JPA  has  become  a generic  one  for 
all  types  of  new  concept  manuals,  as  well  as  for  the  Air  Force's  new  con- 
cept manual. 

FOMM  and  Work  Package  are  terms  used  in  the  Navy  today.  The  basic 
analytic  techniques  and  formats  of  FOMM  are  those  of  SIMM.  The  Army  Job 
Performance  Manual  (JPM)  and  Job  Performance  Guide  (JPG)  are  results  of 
the  study  of  all  other  concepts  plus  some  Army  TEC  program  elements  for 
guiding  on-the-job  training. 

The  Army  specification  MIL-M-632XX  approach  specifically  breaks 
the  compartmentalizing  of  traditional  training  and  technical  documentation. 

The  other  concepts  imply  it  but  are  not  specific. 

There  are  three  other  lines  of  development  that  should  be  discussed 
briefly  at  this  point.  The  first  two  represent  hardware  approaches  to  a 
solution  of  the  maintenance  problem.  The  third  is  a paper  solution  to 
hardware  training  devices  and  hardware  "hands-on"  testing. 

The  first  is  concerned  with  the  medium  of  transmission.  Photographic 
and  electronic  means  of  recording  and  transmitting  information  are  many. 

The  machines  for  storing,  accessing,  displaying,  and  transmitting  informa- 
tion are  being  "sold"  very  hard  by  large  equipment  manufacturers.  There  are 
no  data  that  show  an  improvement  in  performance  or  reduction  in  training 
with  the  use  of  such  media  over  that  of  the  traditional  paper  manual.  The 
arguments  for  it  are  made  primarily  on  the  grounds  of  reduced  storage  area, 
quick  update,  and  the  general  attraction  that  new  machines  have  for  people 
in  our  culture.  The  arguments  used  in  selling  this  equipment  may  be  specious, 
but  we  are  likely  to  see  the  introduction  of  such  equipment  anyway.  There 
is  nothing  in  the  fundamental  elements  that  cannot  be  transmitted  by  any 
medium. 

The  other  entry  of  equipment  producers  is  automated  troubleshooting. 
The  thesis  for  this  equipment  is  that  it  can  replace  the  human.  The  impli- 
cation is  that  the  human  needs  to  be  replaced  because  he  can't  do  the  lob 
as  well  or  costs  more.  Once  we  have  accepted  the  troubleshooting  situation 
as  one  that  is  not  problem-solving  but  can  be  fully  proceduralized,  there 
is  no  reason  that  a machine  can't  be  designed  to  sense  faults — and  even 
to  take  corrective  action.  But  if  a human  being  has  to  be  present  to  take 
any  actions  or  sense  anything,  it  may  be  less  expensive  to  let  him  do  the 
whole  job.  The  man/machine  tradeoff  has  replaced  the  head/book  tradeoff 
as  a critical  question. 

Symbolic  substitutes  are  a derivative  of  the  graphics  in  JPAs. 

Research  is  being  conducted  in  the  Air  Force  and  Army  on  symbolic  substi- 
tutes for  testing  and  for  training.  Graphics  has  been  found  to  mediate 
transfer  to  job  performance  as  well  as  real  equipment  for  performance 
as  well  as  real  equipment  for  most  tasks.  Performance  on  symbolic  substitutes 
correlates  well  with  hands-on  performance  (Shriver  6.  Foley,  1974;  ARI,  in 
press)  and  shows  evidence  of  being  more  valid  than  hands-on,  because  observors 
of  hands-on  performance  tend  to  become  bored  and  inattentive  while  observing 
and  scoring. 
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Impediments  to  Implementation 


The  twin  impediments  to  implementation  have  been  the  institutional 
schools  and  the  institutional  specifications  for  technical  documentation. 
These  institutions  represented  the  traditional  philosophy  of  separate  com- 
partments for  training  and  technical  documentation.  What  success  we  have 
had  is  due  to  the  crevices  provided  by  R&D.  The  good  work  of  individual 
scientists  has  produced  effective  materials,  specifications,  and  tests 
that  showed  good  results  in  operational  cerms,  but  all  of  this  would  have 
remained  in  a backwater  if  there  had  not  been  new  factors  at  work  in  the 
military  establishments.  The  new  factors  are  increased  personnel  costs  and 
a new  philosphy  for  reducing  personnel  costs.  The  military  establishment 
is  ready  to  try  out  new  approaches  to  training  and  technical  documentation 
if  a reasonable  probability  for  reducing  personnel  costs  exists.  The  R&D 
studies  have  been  pulled  together  to  show  there  is  a common  cause  for  their 
good  effects.  The  effects  have  been  costed  out  in  global  terms.  There  is 
more  implementation.  We  are  now  ready  to  consider  what  the  human  resources 
R&D  community  must  do  to  provide  the  basis  for  the  next  thrust. 

We  need  an  overall  framework  or  point  of  view  in  order  to  go  for- 
ward. In  general,  one  might  say  that  implementation  came  slowly  because 
we  did  not  start  out  with  a large-scale  point  of  view  and  build  our  options 
within  that  framework.  We  did  small-scale  studies  that  vied  with  each  other 
for  implementation.  Times  have  changed;  the  framework  has  emerged  from 
our  individual  efforts  along  with  the  outline  of  the  large-scale  cost 
factor  for  personnel. 

The  implementation  of  what  has  been  accomplished  in  20  years  of 
research  has  created  a new  baseline  for  research  that  needs  to  be  done. 
Research  should  not  follow  the  structure  that  existed  in  the  past  or  it 
will  result  in  repetitions  of  what  has  been  done — more  refined  perhaps 
but  not  adapted  to  answer  the  new  questions  that  require  answers. 

New  Directions  for  Information  Transfer  Documentation 


The  basis  of  JPA-type  documentation  is  front-end  analysis  to  make  JPAs 
user-centered,  plus  processes  for  making  the  document  intelligible  for  grade 
school  reading  capability.  This  type  of  analysis  is  what  Instructional 
System  Development  (ISD)  calls  for  but  does  not  make  explicit.  ISD  has  had 
wholesale  adoption.  This  has  opened  doors  that  were  closed.  ISD  calls 
for  decisions  to  be  made  by  training  analysts  that  previously  were  the 
perogative  of  the  institutional  instructor,  but  it  provides  no  specifica- 
tion of  techniques  for  making  these  decisions.  The  Army's  Improved  Tech- 
nical Documentation  and  Training  specification  (ITDT  MIL-M-632XX)  recognizes 
the  commonality  of  Technical  Documentation  and  Training.  The  objective  is 
to  impart  information  that  increases  the  capability  of  personnel  to  do  tasks. 
The  JPA  represents  training,  criterion-referenced  tests,  and  training  devices 
Just  as  much  as  it  does  technical  documentation.  Another  important  point 
about  documented  information  is  that  it  is  not  limited  to  an  institutional 
site;  it  is  moveable  to  the  job  site. 
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The  difficulties  of  the  past  are  past.  The  problem  for  the  future  is 
developing  the  potential  of  documentation  for  increasing  user  capabilities 
Research  must  move  quickly  to  fill  the  space  which  has  been  made  available 
with  the  adoption  of  ISD.  The  research  payoff  for  the  next  few  years  will 
be  in: 


1.  Technical  documentation  as  training  simulators  and  tests  to  make 
OJT  effective. 

2.  Technical  documentation  as  a means  to  expand  job  duties  each  person 
can  perform — first  and  successive  enlistments. 

3.  How  much  to  proceduralize  troubleshooting  for  maximum  job  pro- 
ductivity . 

4.  Cost  effectiveness  studies  in  a simulation  model  that  continually 
updates  personnel  capabilities  versus  machine  and  identifies  what  capabili- 
ties in  personnel  will  produce  the  greatest  cost  effectiveness  benefit. 

Research  is  not  needed  on  the  old  problem  like  which  new  concept  is 
best  or  which  format  is  best  for  which  situation,  or  the  head/book  trade- 
off, or  making  formats  so  experienced  personnel  like  them.  Those  issues 
have  been  washed  away  with  the  opening  of  the  flood  gates.  We  must  immedi- 
ately get  to  work  on  the  problems  that  have  major  importance. 

Institutional  training  will  be  drastically  reduced  because  it  is  high 
cost,  relatively  ineffective,  and  not  in  vogue.  Training  at  the  job  site 
will  be  the  dominant  theme,  and  documentation  makes  OJT  possible  without 
turning  the  supervisor  into  an  instructor.  But  the  cost  effectiveness  of 
alternatives  must  be  established  in  the  next  few  years,  and  either  the  pen- 
dulum will  return,  or  troubleshooting  by  machines  will  fill  the  gap.  We 
have  to  understand  the  commonality  of  technical  documentation,  training, 
training  devices,  and  criterion-referenced  tests.  They  all  spring  from 
the  same  analytic  roots.  The  roots  are  front-end  analysis.  The  front-end 
analyses  developed  for  new  concepts  in  maintenance  documentation  represent 
the  furthest  advancement  in  the  state-of-the-art  and  the  best  documented 
source  of  guidance  for  achieving  reductions  in  personnel  costs  and  in- 
creased system  effectiveness.  Training  and  technical  manuals  are  no  longer 
stand-alone  processes.  They  work  together  in  a process  that  includes  train- 
ing devices  and  performance  testing  as  well. 

The  research  personnel  who  focus  on  technical  documentation  must  recog- 
nize what  they  have  and  where  it  fits.  They  have  the  most  powerful  analytic 
tools  and  the  best  means  for  delivering  information  to  users  on  the  job  as 
required  by  the  ISD  process.  They  must  quickly  learn  to  perceive  what  new 
effects  can  be  produced  with  these  tools  and  delivery  devices.  For  instance, 
one  high-payoff  effect  is  that  personnel  trained  on  the  job  produce  useful 
products  while  learning  a wider  variety  of  Job  task  performance.  The  indi- 
vidual can  continue  learning  more  and  different  tasks  far  beyond  existing 
boundaries  of  job  assignments  based  on  old  assumptions  about  training  costs. 


We  must  preapre  to  answer  questions  on  the  payoff  of  expanded  job  capabilities 
and  the  payoff  of  increased  productivity.  The  question  is  no  longer  insti- 
tutional training  versus  OJT.  It  is  how  the  personnel  subsystem  will  change 
as  a result  of  increased  individual  capabilities  developed  on  the  Job  and 
what  the  payoffs  are  for  these  changes.  The  changes  we  want  to  promote 
are  of  overall  system  importance.  They  are  highly  visible,  large  payoffs. 

Other  research  questions  center  on  the  mechanics  of  OJT.  For  example, 
should  the  new  personnel  be  given  job  tasks  as  they  come  up  in  the  main- 
tenance situation  or  should  they  follow  a prescribed  series  of  tasks  deemed 
appropriate  for  a new  person  on  the  job?  Finally,  the  locus  of  the  research 
should  be  recognized  as  the  job  site.  In  the  Navy,  the  job  site  is  often  a 
floating  platform — relatively  inaccessible  to  research  personnel.  There- 
fore, the  experiments  have  to  be  set  up  to  run  without  the  presence  of  a 
researcher.  This  may  require  some  research  on  remote  control  of  conditions. 

What  are  the  payoff  situations  today? 

1.  Getting  useful  work  from  the  personnel  without  nonproductive  school 
training.  The  payoff  potential  of  this  area  is  on  the  order  ot  $500  million 
dollars  per  year  in  each  service. 

2.  Getting  higher  productivity  from  individuals  on  the  job  from  the 
first  day  on.  The  payoff  potential  of  this  area  is  on  the  order  of  $1 
billion  dollars  per  year  in  each  service. 

3.  Expansion  of  job  duties  and  career  development.  The  payoff  potential 
of  this  area  overlaps  the  previous  two  to  some  extent  but  probably  repre- 
sents an  additional  savings  on  the  order  of  $1-1/2  billion  dollars  per  year 
for  each  service,  and  a higher  readiness  state  than  the  services  have  yet 
experienced. 

4.  Models  of  systems  operations  are  needed: 

a.  To  document  increases  in  readiness  brought  about  by  new  effects. 

b.  To  prioritize  research  by  "what  if"  studies. 

c.  To  conduct  man/machine  tradeoffs. 

d.  To  manage  personnel  operations  (including  imbalance  of  forces). 

e.  To  codify  personnel  operations  so  they  can  be  traded  off 
against  hardware  developments  in  top-level  discussions. 

In  doing  this,  we  have  to  take  a new  view  of  training,  training  de- 
vices, manuals,  OJT,  job  structure,  career  patterns,  performance  testing, 
and  manning  levels.  We  have  to  get  in  tune  with  operational  and  system 
design  questions  for  structuring  our  research  questions.  The  human  re- 
sources community  has  marched  in  this  direction  before.  We  created  Qualita- 
tive Quantitative  Personnel  Requirements  Information  (QQPRI)  and  human  factors 
specialists  in  the  process,  but  if  we  are  honest  with  ourselves  we  will  look 
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around  us  and  say  that  the  hardware  community  still  owns  the  battlefield. 

We  lost,  but  we  did  gain  an  identity  and  experience  by  fighting  the  battle. 
Now  let  us  use  whatever  wisdom  and  experience  the  "old  heads"  can  bring  to 
bear  to  structure  the  new  research  efforts.  There  are  few  "old  heads,"  and 
they  have  cracked  against  each  other  enough  that  they  are  willing  to  opt 
for  cooperation  rather  than  conflict  with  each  other.  There  is  some  fund- 
ing. There  is  some  new  blood.  There  is  enough  difference  in  viewpoint  to 
engage  in  several  efforts  at  once  that  contribute  to  each  other  in  a common 
frame  of  reference.  The  human  resources  research  community  must  not  squander 
its  resources  in  fighting  or  in  repeating  the  past. 

It  is  with  this  background  that  I suggest  some  specifics  in  line  with 
the  general  areas  outlined  above. 

1.  Project  Area  1.  A computer-based  effectiveness  model  for  tradeoff 
studies  is  needed  to  identify  specifically  how  having  personnel  with  certain 
kinds  of  technical  information  will  effect  the  cost  and  readiness  of  the 
service.  This  is  a "what  if"  model  to  define  research  alternatives,  to 
establish  research  priorities,  and  to  provide  the  detailed  rationales  and 
arguments  for  the  use  of  personnel  components  and  for  the  use  of  certain 
hardware  components.  It  must  be  an  iterative  model,  not  a "one  shot" 
model  that  comes  up  with  personnel  demands  which  the  hardware  must  meet. 

In  addition  to  all  the  personnel  factors,  it  must  model  the  hardware  side 
as  well  as  training,  testing,  career  structure  manuals,  training  devices, 
and  other  "human  factors."  Its  output  effectiveness  must  be  in  material 
readiness  to  do  the  job,  not  personnel  errors  or  personnel  tasks,  skills, 
or  knowledges.  The  overall  results  must  be  in  terms  of  how  ready  the  man/ 
machine  system  is  to  do  its  job.  On  the  personnel  side,  this  will  reduce 
to  productivity  and  from  that  into  tasks,  skills,  training,  etc. 

A cost  model  must  be  developed  to  go  with  the  effectiveness  model. 

The  cost  model  must  include  both  the  personnel  and  hardware  side  in  gross 
terms  and  trace  the  personnel  side  down  into  contributory  factors  of  train- 
ing, skill,  tasks,  etc. 

2.  Project  Area  2.  Development  of  methods  for  shaping  personnel  char- 
acteristics throughout  a career.  This  includes  getting  the  new  personnel 
job-effective  quickly,  expanding  their  skills  to  increase  their  generality 
in  successive  enlistments,  and  determining  how  this  ties  back  into  life 
cycle  costing  of  the  cost-effectiveness,  "what  if"  model.  Career  testing 
and  promotion  are  included. 

3.  Project  Area  3.  An  information  transfer  project  with  training  and 

manuals  and  other  media  such  as  training  devices.  "Manuals"  should  be  ex- 
panded to  include  other  media.  Training  devices  and  manuals  should  be  an 
integrated  package:  OJT,  self-paced  instruction  learning  from  the  JPA, 

school  training  for  2nd  and  3rd  termers,  expansion  into  broader  fields 
(e.g.,  from  Radar  A to  all  radars  to  all  electronic  equipment  to  adminis- 
tration and  supply  of  all  electronics).  This  is  an  area  where  the  overall 
model  can  breakout  those  factors  which  enable  personnel  to  do  different 
things  that  the  larger  model  Indicates  are  cost  effective. 
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1. 

4.  Project  Area  4.  Personnel  logistics.  This  area  should  concentrate 
on  what  equipment  manufacturers  are  doing  in  automated  troubleshooting  equip- 
ment, spares  provisioning,  transportation  of  spares,  storage  of  spares, 
costs  of  parts  transportation,  tactical  operations,  etc.  It  is  a breakout 
from  Project  Area  1 and,  like  the  other  areas,  feeds  information  back  into 
the  model  of  Area  1. 

5.  Project  Area  5.  Personnel  System  Architecture.  The  "front  men" 
for  the  personnel  subsystem  in  the  system  design  and  life  cycle  should  be 
working  in  this  area.  It  incorporates  all  that  is  developed  from  the  other 
areas,  but  is  different  in  that  this  group  must  also  "go  beyond"  what  is 
known.  They  have  to  give  "best  guesses"  during  concept  development  stages 
and  then  identify  the  studies  needed  to  detail  out  the  best  guesses.  These 

j personnel  include  the  most  senior  personnel  available  in  the  human  resources 

research  field.  There  will  be  Ad  Hoc  Teams,  but  a basic  cadre  is  also  needed. 
Staffing  will  include  senior  civil  service  and  contractor  personnel.  Of  all 
the  areas,  this  is  the  most  unusual  because  it  goes  beyond  research  into 
implementation.  It  uses  all  that  is  known,  adds  the  best  guesses  to  solve 
the  new  problem,  and  defines  what  studies  are  needed  to  iterate  the  research 
process.  It  will  require  a lot  more  thought  than  offered  here.  Project 
managers  for  particular  personnel  subsystems  may  be  required.  However, 
this  area  is  the  spearhead  for  implementation  of  research.  Without  it, 
the  research  may  be  filed  in  the  "archives  of  arcane  results."  This  spear- 
head also  establishes  the  need  for  research.  These  personnel  are  engaged 
in  the  operational  world,  where  research  input  goes  in  and  research  needs 
come  back.  It  makes  the  human  resources  research  community  far  more  visible 
in  the  operational  world  than  it  has  been  heretofore,  but  it  requires  re- 
search personnel  to  "go  beyond"  their  data.  This  has  always  been  anathema 
to  university-trained  experimenters.  The  anathema  hasn't  stopped  researchers 
from  going  beyond  their  data  in  the  past.  We  have  all  done  it  many  times, 
but  we  have  done  it  from  a very  restricted  frame  of  reference.  Our  research 
conditions  have  seldom  matched  the  operational  conditions  they  were  supposed 
to  predict,  but  that  didn't  stop  us,  and  it  didn't  mean  we  were  wrong. 
Basically,  we  have  been  right,  and  the  world  has  come  into  congruence 
with  us  in  large  measure.  Now,  as  we  start  out  from  this  new  baseline, 
we  must  create  the  operational  conditions  of  the  real-world  research,  then 
break  out  to  apply  it  to  major  practical  problems.  Above  all,  we  must  make 
the  jump  from  our  new  framework  of  conditions  to  the  operational  world  with 
best  guesses.  The  jump  will  be  shorter  than  it  was  in  the  past,  but  this 
time  we  must  say  we  will  make  the  jump  and  build  the  platform  to  jump  from. 
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IMPLEMENTATION  OF  THE  JPA/JOB-ORIENTED  TRAINING  APPROACH 
TO  MAINTENANCE:  THE  IMPACT  ON  PERSONNEL  SYSTEMS 

John  J.  Klesch 

Air  Force  Human  Resources  Laboratory 

Introduction 

Innovations  being  made  in  the  Department  of  Defense  in  the  areas  of 
technical  training  and  technical  manuals  for  maintenance  can  produce  sig- 
nificant savings  in  the  life  cycle  cost  of  weapons  systems.  These  innova- 
tions can  also  produce  negative  side  effects  that,  if  not  solved  by  tech- 
nological or  administrative  changes,  can  seriously  reduce  their  effectiveness. 
One  such  area  of  impact  is  the  personnel  system.  Personnel  specialists 
show  considerable  interest  in  the  potential  of  these  techniques  for  reducing 
aptitude  requirements  and  thus  expanding  the  manpower  pool.  They  also  are 
interested  in  facilitating  the  transfer  of  maintenance  personnel  between 
systems  without  extensive  formal  training.  At  the  same  time,  they  show 
concern  for  the  impact  these  techniques  will  have  on  skill  upgrading,  re- 
cruitment, and  retention.  Although  the  Air  Force  has  not  conducted  any 
in-depth  studies  of  the  impact  of  personnel  policies  of  implementation  of 
these  techniques,  a number  of  studies  on  Job-oriented  Training  and  JPAs 
have  been  performed  that  can  be  used  to  deduce  their  effects,  and  like- 
wise, to  suggest  some  changes  that  will  have  to  be  made  to  present  person- 
nel policies  in  order  to  accommodate  these  techniques. 

Job-oriented  Training  Studies 

Three  major  studies  on  Job-oriented  Training  have  been  performed  which 
have  produced  technicians  who  entered  the  Air  Force  maintenance  environment. 
These  studies  were  all  designed  to  determine  if  a significant  reduction  in 
entry-level  training  is  practical.  The  approach  taken  in  each  course  de- 
velopment was  to  eliminate  or  reduce  broad-based  principles  and  concentrate 
on  hands-on  equipment  training  using  essentially  rote  maintenance  techniques. 
The  students  learn  to  perform  specific  job  tasks  by  following  carefully  pre- 
pared step-by-step  instructions.  Although  JPAs  were  not  utilized  in  these 
efforts,  it  should  be  evident  that  a JPA/Job-oriented  approach  would  be 
very  similar.  It  would  be  expected  that  subsequent  performance  in  the 
field  would  be  improved  if  JPAs  were  available.  This  obvious  difference 
does  not  negate  the  findings  reported  here,  however,  since  the  problems 
encountered  involved  variables  other  than  routine  task  performance. 

The  Communications-Electronics  Service  Test 

This  study  (DoD  & USAF,  1970),  which  was  begun  in  1966,  involved 
the  application  of  two  different  concepts  to  reduce  training:  the  job- 

oriented  or  "X"  approach  and  a pr inc iples-only  or  "Y"  approach.  Both  con- 
cepts were  applied  to  two  courses:  an  airborne  electronic  equipment  re- 

pair course  and  a ground  electronics  repair  course.  The  "C"  conventional 
courses  were  38  weeks  and  42  weeks  in  length,  respectively.  Application 
of  the  concepts  resulted  in  approximately  a 40  percent  reduction  in  course 
length  for  both  the  X and  Y approaches.  A total  of  735  subjects  partici- 
pated (294  X graduates,  294  C graduates,  and  147  Y graduates).  Questionnaires, 


surveys,  and  supervisory  appraisals  were  utilized  to  obtain  data  on  the 
subjects'  performance  and  problems  throughout  their  initial  4-year  tour. 
Although  a number  of  criticisms  could  be  made  of  the  methodology  employed 
in  the  study,  the  types  of  problems  encountered  are  nonetheless  valid  and 
must  be  addressed  in  any  future  applications. 

Performance. 

1.  Results . The  X graduates  demonstrated  a performance  capability 
equal  to  and  sometimes  greater  than  the  C students  during  the  first  10  weeks 
on  the  job.  The  Y graduates  were  less  proficient,  particularly  in  the  use 
of  test  equipment  and  published  maintenance  procedures.  Overall  job  per- 
formance ratings  by  supervisors  at  the  first-year  point  indicated  that  the 

C and  Y graduates  were  rated  significantly  higher  than  the  X graduates.  The 
X graduates  were  not  as  effective  or  flexible  in  adpating  to  new  and  un- 
familiar maintenance  demands.  When  confronted  by  unfamiliar  equipment  or 
unorthodox  troubles,  the  X graduates  frequently  needed  extensive  assistance. 
While  the  Y students  encountered  similar  problems,  they  appeared  to  profit 
more  readily  from  experience  and  further  training.  The  X graduates  were 
given  fundamentals  training  during  this  period  but  encountered  difficulty 
absorbing  and  applying  this  training.  At  the  conclusion  of  the  4-year  tour, 
the  supervisors  rated  the  overall  job  performance  of  all  graduates  about 
equal.  On  specific  job  performance  factors,  however,  supervisors  consis- 
tently rated  a higher  percentage  of  the  X graduates  unsatisfactory  than 
either  the  Y or  C graduates. 

2.  Conclusions.  Specific  job  task  training  results  in  immediate 
task  proficiency  but  the  transfer  of  these  skills  to  other  tasks,  parti- 
cularly in  unorthodox  situations,  cannot  be  assumed.  Supervisors  (conven- 
tionally trained)  did  not  perceive  the  X graduates  to  be  the  equivalent 

of  the  C graduates  but  believed  the  Y graduates  were  closer  to  the  C gradu- 
ates. 


3.  Implications.  Job  Performance  Aids  will  have  to  be  available 
for  all  systems  that  X-type  graduates  are  assigned  to  repair.  Training 
and/or  expert  assistance  will  be  required  to  handle  special  problems. 

Mixture  of  conventionally  trained  personnel  with  X graduates  in  the  same 
grade  level  will  probably  result  in  better  ratings  being  awarded  to  the 
former  group. 

Training. 

1.  Results.  At  the  1-year  and  2-1/2-year  points,  nearly  50  percent 
of  the  X graduates  felt  that  their  initial  training  was  inadequate  compared 
to  only  7 percent  of  the  C graduates.  While  41  percent  of  the  Y graduates 
indicated  inadequacy  of  training  at  the  1-year  point,  this  later  dropped  to 
29  percent.  All  graduates  felt  that  they  were  able  to  attain  satisfactory 
proficiency  in  their  normal  job  routines  within  a reasonable  period  of  OJT. 
Over  50  percent  of  the  X and  Y graduates  rated  their  initial  training  as  in- 
adequate for  preparing  them  for  their  career  development  course  (CDC)  pro- 
gram. However,  supervisors  rated  all  graduates  approximately  equal  in  "speed 
of  completing  CDC"  and  "overall  performance  as  CDC  students."  Over  50  percent 
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of  the  X graduates  rated  their  training  as  inadequate  in  preparing  for 
the  Speciality  Knowledge  Test  (SKT) , compared  to  37  percent  of  the  Y gradu- 
ates and  6 percent  of  the  C graduates.  At  the  1-year  point,  69  percent  of 
the  X graduates  had  passed  the  five  skill  level  SKT  compared  to  75  percent 
and  83  percent  for  the  Y and  C graduates  respectively.  C graduates  attained 
the  five  skill  level  an  average  of  3 weeks  sooner  than  the  X or  Y graduates. 

A substantial  amount  of  additional  fundamentals  training  was  provided  to 
the  X graduates  in  preparing  them  for  upgrading  to  the  five  level  compared 
to  the  Y and  C graduates. 

2.  Conclusions . Substantial  additional  training  was  required  to 
prepare  X graduates  for  skill  upgrading.  Initial  training  was  felt  to  be 
inadequate  for  both  CDC  progression  and  SKTs. 

3.  Implications.  Careful  study  of  task  requirements  at  the  five 
skill  level  need  to  be  made  to  ensure  that  proper  training  and/or  JPAs  are 
provided.  In  the  above  system,  it  appeared  that  there  were  many  tasks  that 
the  X graduates  could  not  perform  without  considerable  additional  training. 

It  is  unlikely  that  such  an  approach  will  be  cost  effective,  even  though 
initial  training  has  been  reduced.  In  order  for  X training  to  be  viable,  the 
field  requirements  for  the  graduates  would  have  to  be  carefully  controlled 

or  changed.  Examples  of  changing  field  task  requirements  would  be:  re- 

stricting troubleshooting  to  black  box  replacement,  providing  better  trouble- 
shooting aids  to  enable  the  technician  to  troubleshoot  to  subassembly  or 
module  level,  or  providing  automatic  test  equipment.  The  present  personnel 
system  operated  against  the  X approach  in  that  the  graduates  were  assumed  to 
have  all  of  the  knowledges  of  the  conventional  graduate.  It  also  was 
assumed  that  these  knowledges  were,  in  fact,  required  to  perform  the  tasks. 
Future  efforts  should  examine  this  notion  more  carefully.  The  career  devel- 
opment courses  and  speciality  knowledge  tests  would  also  require  substantial 
revision.  In  effect,  new  specialities  would  have  to  be  created.  Such  a 
speciality  need  not  be  restricted  to  the  repair  of  one  equipment  so  long 
as  JPAs  and  the  same  field  requirements  are  available. 

Other . 

1.  Results.  At  the  conclusion  of  their  initial  tour,  87  percent 
of  the  X and  Y graduates  had  been  promoted  to  E-4  compared  to  92  percent 
of  the  C graduates.  The  attitude  and  morale  of  the  X and  Y graduates  were 
judged  to  be  favorable  even  though  they  had  judged  their  training  to  be 
unsatisfactory.  Supervisors  indicated  a general  acceptance  of  the  graduates 
but  remained  highly  critical  of  the  X and  Y training  concepts.  Job  assign- 
ments and  utilization  of  experimental  graduates  were  restricted,  particularly 
in  the  airborne  specialty.  Supervisors  stated  that  they  were  forced  to 
restrict  the  duties  of  these  graduates  to  the  flightline  because  of  their 
inability  to  perform  the  more  complex  tasks.  There  was  considerable  doubt 
that  the  experimental  graduates  would  have  achieved  an  acceptable  level  of 
job  proficiency  if  there  had  not  been  qualified  electronic  technicians, 
supervisors,  and  trainers  to  provide  detailed  assistance.  After  3 years 
of  service,  approximately  the  same  percentage  of  all  three  types  of  gradu- 
ates indicated  their  intention  to  reenlist  (11%). 


2.  Conclusions . Neither  promotion  nor  reenlistment  rate  appeared 
to  be  seriously  affected  by  the  initial  type  of  training  received.  However, 
job  assignments  and  utilization  appear  to  have  been  severely  restricted. 
Supervisors  had  been  and  remained  critical  of  the  experimental  courses. 

Despite  promotion  actions  and  ratings  of  overall  satisfactory  performance, 
the  experimental  graduates  were  considered  to  be  less  capable  than  the 
conventionally  trained  graduates. 

3.  Implicat ions . If  no  further  training  is  provided,  airmen  are 
not  likely  to  pass  their  SKT  and  hence  will  not  get  promoted.  In  this  study, 
they  were  promoted  but  presumably  only  because  of  further  intensive  training 
that  was  provided  in  their  field.  As  noted  earlier,  this  approach  is  prob- 
ably not  cost  effective.  The  restricted  work  assignments  given  to  the  X 
graduates  implies  that  jobs  may  have  to  be  restructured  to  accommodate  the 
utilization  of  these  personnel.  The  restriction  may  also  have  been  made 

on  the  basis  of  bias  on  the  part  of  the  supervisor.  While  that  may  be  true, 
it  is  not  likely  to  be  overcome  so  long  as  conventionally  trained  graduates 
are  mixed  with  the  experimentally  trained  graduates  in  a given  speciality. 

The  reenlistment  rate  for  all  groups  indicates  that  this  personnel  problem 
must  be  addressed  in  any  solutions  that  are  fomulated.  Offering  career 
resident  training  at  the  3-year  point  may  have  provided  an  incentive  for 
more  of  the  X and  Y graduates  to  reenlist. 

The  Learner-Centered  Instruction  (LCI)  Study 

In  this  study  (Pieper,  Sweezey,  & Valverde,  1970),  a Job-oriented 
Training  approach  (called  Learner -Centered  Instruction)  was  used  for  the 
F-11A  Weapons  Control  Systems  Mechanic  Course.  The  conventional  course 
was  24  weeks  in  length:  10  weeks  of  electronic  fundamentals  and  14  weeks 

of  principles-centered  instruction  on  equipment  sets.  The  LCI  course  was 
14  weeks  in  length.  This  was  a reduction  of  close  to  60  percent  of  the 
original  course.  Eighty  airmen  trainees  received  the  LCI  course;  40  had 
the  required  high  electronics  aptitudes  and  40  had  medium  electronics  aptitudes 
and  would  not  normally  be  utilized  in  this  specialty.  The  subjects  were 
matched  with  40  airmen  who  received  the  regular  course  training.  Special 
tests  were  constructed  to  measure  the  graduates'  performance  ability  and 
knowledge  at  both  the  end  of  training  and  after  5 months  on  the  job.  Super- 
visory ratings  and  student  ratings  were  also  obtained.  Anecdotal,  but 
nevertheless  important,  events  occurring  after  the  evaluation  are  reported 
here  because  of  their  relevance  to  the  topic  of  this  paper.  The  program 
was  conducted  from  1968  to  1969. 

Performance . 

1.  Results . The  high-aptitude  LCI  graduates  were  superior  to  the  C 
graduates  at  both  the  end-of-course  test  and  after  5 months  on  the  job  using 
job  sample  tests  as  the  criterion  measure.  The  medium  aptitude  LCI  gradu- 
ates were  very  similar  in  performance  capability  to  the  C graduates.  The  C 
graduates  performed  better  than  the  LCI  graduates  on  a "Practical  Test" 
that  is  normally  administered  at  the  end  of  the  C course.  Special  perfor- 
mance-oriented ratings  obtained  from  supervisors  at  the  5-month  period  in- 
dicated no  consistent  differences  among  the  three  groups  in  ability  to  per- 
form the  Job. 
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2.  Conclusions.  The  results  again  indicate  that  teaching  specific 
task  performance  results  in  superior  task  performance,  especially  when  measured 
by  job  sample  tests.  Since  the  LCI  graduates  did  not  fare  as  well  on  the 
"Practical  Tests"  though,  the  doubt  remains  as  to  whether  the  specific  train- 
ing is  readily  transferred  to  other  tasks.  This  type  of  training  appears  to 
permit  the  utilization  of  medium  aptitude  personnel  to  perform  such  tasks. 
Supervisory  ratings  indicate  that  these  skills  are  retained  on  the  job. 

3.  Implications.  If  specialities  can  be  designed  that  are  restricted 
to  the  kinds  of  tasks  measured  by  the  job  sample  tests,  it  will  be  quite 
feasible  to  expand  the  manpower  pool  by  the  utilization  of  medium  aptitude 
personnel. 

Training. 

1.  Results . The  C students  were  superior  to  the  LCI  graduates  in 
performance  on  a specially  designed  knowledge  test  (similar  to  the  Speciality 
Knowledge  Test  used  for  promotion  and  skill  upgrading)  at  both  the  end-of- 
course  and  5-month  intervals.  Students  indicated  a general  satisfaction 
with  the  training  at  the  end  of  the  course;  however,  no  measurement  of 
satisfaction  was  made  at  the  5-month  interval. 

2.  Conclusions . It  is  unfortunate  that  no  measurements  of  student 
satisfaction  were  taken  at  the  5-month  point  since  the  speciality  test  re- 
sults indicate  that  they  would  have  a difficult  time  passing  the  Speciality 
Knowledge  Test  without  considerable  additional  training  in  theory.  The  C 
course  was  obviously  designed  to  provide  this  knowledge. 

3.  Implications.  Unless  a new  speciality  is  created  along  with 
new  Career  Development  Courses  and  Speciality  Knowledge  Tests,  considerable 
further  special  training  for  LCI  graduates  will  be  required.  Normal  OJT 
and  career  development  courses  are  not  likely  to  be  sufficient  to  provide 
the  knowledge  of  fundamentals  expected  in  this  speciality. 

Other. 

1.  Results . The  following  anecdotal  comments  are  not  documented 
in  the  study  reported  above  but  are  important  events  that  occurred  after 
the  original  study  was  concluded.  Field  personnel  reported  that  LCI 
graduates  performed  satisfactorily  at  the  three  skill  level  but  that  per- 
formance had  been  below  average  at  the  five  skill  level,  presumably  because 
of  a lack  of  knowledge  and  basic  fundamentals.  LCI  graduates  were  reported 
as  feeling  in  an  inferior  position  to  their  contemporaries.  The  work  load 
on  supervisors  and  OJT  trainers  was  greatly  increased.  Time  required  to 
upgrade  the  LCI  personnel  to  the  five  skill  level  was  about  4 months  longer 
than  normal. 

2.  Conclusions . Froblems  in  morale,  additional  training  workloads, 
delays  in  upgrading  and  promotions,  and  a negative  attitude  toward  reenlist- 
ment are  all  likely  to  have  resulted  from  the  present  personnel  system. 

Since  five  skill  level  performance  tests  were  not  utilized  as  they  were  at 
the  three  skill  level,  it  is  not  certain  as  to  whether  or  not  actual  per- 
formance deficiencies  surfaced.  The  LCI  graduates'  inability  to  readily 


use  existing  CDC  materials  and  pass  the  SKT  tests  may  have  been  considered 
as  the  performance  deficiencies.  If  the  LCI  graduates  were  never  permitted 
to  perform  five  skill  level  tasks  because  the  supervisors  felt  they  were 


not  qualified  (on  the  basis  of  the  CDC  and  SKT  square  filling  personnel 
exercise),  then  this  should  not  be  taken  as  an  indication  of  a performance 
deficiency. 

3.  Implications.  Future  efforts  must  deal  with  this  dilemma  head- 
on.  The  problem  of  possible  bias  must  be  dealt  with  directly  by  permitting 
new  concept  trainees  to  perform  all  of  the  tasks  required.  Personnel  poli- 
cies of  the  square  filling  variety  that  govern  skill  upgrading  (and  hence 
permit  the  technician  to  perform  the  task)  should  be  restricted  to  actual 
performance  tests  so  that  the  supervisor  can  be  assured  that  the  technician 
indeed  can  perform  the  task  assigned. 

Project  Hasty  Chief 

This  project  began  in  1976  and  involved  the  application  of  Instruc- 
tional Systems  Development  procedures  to  an  aircraft  maintenance  specialty. 
The  conventional  course  is  12  weeks  in  length  and  contains  broad-based  prin- 
ciples training  as  well  as  some  practical  training.  It  is  designed  to  pre- 
pare technicians  to  maintain  a variety  of  aircraft.  The  technicians  are  then 
assigned  to  a Field  Training  Detachment  (FTD)  school  for  specific  training 
on  the  aircraft  to  which  they  are  assigned.  This  FTD  course  is  usually  4 
weeks  in  length  and  contains  principles  training  for  the  specific  aircraft. 
The  Hasty  Chief  initial  training  is  4 weeks  in  length  and  includes  a general 
aircraft  introduction,  safety  instruction,  supply,  and  maintenance  data  re- 
porting procedures.  The  technician  is  then  assigned  to  a FTD  for  4 weeks 
of  hands-on  training  on  the  specific  aircraft.  Half  of  each  day  is  spent 
in  the  classroom  and  the  remaining  half  is  spent  on  the  flight  line  per- 
forming the  tasks  presented  in  the  classroom.  Thus,  although  total  training 
time  has  been  reduced  by  50  percent,  the  graduates  should  be  more  proficient 
in  specific  task  performance.  A total  of  96  students  have  been  entered  into 
this  training. 

1.  Results . No  specific  job  task  performance  tests  were  used  to 
compare  the  performance  of  Hasty  Chief  graduates  with  conventional  gradu- 
ates. At  this  time,  only  end-of-course  interview  results  and  preliminary 
field  ratings  are  available.  Hasty  Chief  graduates  performed  the  specific 
tasks  as  well  as  conventional  graduates.  Using  commands  have  indicated 
acceptance  of  the  graduates  to  date,  although  they  expressed  some  reserva- 
tion about  the  large  amount  of  equipment  required  to  support  the  job- 
oriented  training.  At  the  present  time,  there  are  no  plans  to  revise 
either  the  Career  Development  Course  or  the  Speciality  Knowledge  Test. 

Results  of  these  events  concerning  skill  upgrading  and  future  performance 
will  have  to  be  obtained  before  definitive  conclusions  can  be  made. 

2.  Implications.  Many  of  the  mechanical  tasks  of  this  speciality 
are  straightforward,  such  as  servicing,  towing,  and  inspecting,  and  may 

be  particularly  suitable  to  the  hands-on  training  approach.  Nevertheless, 
the  future  performance  and  skill  progression  of  the  Hasty  Chief  graduates 
will  be  of  prime  importance  to  the  successful  implementation  of  this  con- 
cept. The  refusal  of  the  personnel  people  to  modify  or  eliminate  the  SKT 
may  cause  problems  similar  to  those  encountered  by  the  X graduates  and  LCI 
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graduates.  Failure  or  delay  in  receiving  skill  upgrading  will  then  be  met 
by  a frustrating  attempt  to  provide  fundamentals  training  in  the  field. 
Current  plans  are  to  provide  advanced  resident  training  to  those  techni- 
cians who  commit  to  a career  beyond  the  first  enlistment.  While  this 
approach  sounds  good  in  theory,  the  potential  problems  that  could  arise 
in  the  meanwhile  could  negate  the  whole  purpose  of  the  exercise — namely, 
to  obtain  more  productive  time  out  of  the  first-term  airmen.  In  addition, 
if  the  SKTs  prove  to  be  a stumbling  block  to  skill  level  upgrading  and  pro- 
motion, morale  and,  hence,  retention  problems  may  be  so  damaging  that  the 
additional  training  inducement  will  lose  its  effect. 

Job  Performance  Aid  Studies 

Three  efforts  will  be  briefly  summarized  here:  a demonstration  of  the 

JPA/ Job-oriented  Training  concept,  a controlled  experimental  test,  and  a 
JPA  field  impact  study.  While  none  of  these  studies  addressed  personnel 
system  problems  directly,  they  are  reported  here  to  provide  insight  and 
possible  hypotheses  that  will  be  useful  in  planning  future  efforts. 

Demonstration  of  a JPA/Job-oriented  Approach 

This  study  (Mullen  & Joyce,  1974)  involved  the  development  of  JPAs 
and  Job-oriented  Training  for  a complex  electronic  system.  Eight  high  ap- 
titude and  eight  medium  aptitude  graduates  of  basic  training  were  used  as 
subjects.  No  training  in  electronic  fundamentals  was  provided.  Subjects 
were  taught  soldering,  test  equipment  use,  equipment  geography  and  nomen- 
clature, and  use  of  JPAs.  JPAs  were  provided  that  would  permit  the  tech- 
nicians to  perform  both  flightline  and  shop  tasks.  At  the  end  of  the  course, 
a demonstration/quasi-evaluation  was  made  during  which  the  students  were 
presented  a variety  of  tasks  to  perform.  The  students  were  then  reassigned 
to  conventional  technical  schools  in  a variety  of  specialties.  The  course 
took  about  4 weeks,  compared  to  the  normal  35-week  course. 

1.  Results^.  Roth  groups  were  able  to  successfully  complete  the 
course  that  utilized  proficiency  tests  as  a criterion  for  course  completion. 
High  aptitude  personnel  were  superior  to  medium  aptitude  personnel.  Results 
of  the  demonstration  tests  were  disappointing.  High  aptitude  personnel 
again  were  superior  to  medium  aptitude  personnel;  both  groups,  however, 
made  errors.  Skills  taught  in  the  course  were  performed  satisfactorily. 

Using  the  JPAs  to  perform  troubleshooting  tasks,  both  groups  could  not  find 
the  majority  of  problems  without  assistance.  Under  controlled  experimental 
conditions,  it  is  safe  to  say  they  would  have  failed.  While  part  of  the 
performance  problem  could  be  directly  attributable  to  the  stressful  nature 
of  the  observed  demonstration,  the  most  readily  identified  problems  were 

in  the  JPAs  themselves.  The  JPAs  had  a number  of  technical  errors  but, 
more  importantly,  apparently  could  not  adequately  convey  the  information 
in  a manner  in  which  the  specially  trained  technicians  could  use  it  effec- 
tively. 

2.  Conclusions . The  Job-oriented  Training  portion  of  this  study 
indicated  that  fundamentals  training  was  not  reqxiired  to  obtain  certain  types 
of  task  performance.  The  graduates  were  unable,  however,  to  circumvent 
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problems  presented  when  the  JPAs  were  inadequate.  Medium  aptitude 
personnel  had  a more  difficult  time  than  did  the  high  aptitude  per- 
sonnel . 


3.  Implications.  This  approach  to  maintenance  will  be  entirely 
dependent  upon  the  ability  to  produce  good  JPAs.  If  the  graduates  are 
restricted  to  only  some  of  the  tasks  and  the  other  tasks  are  not  changed, 
two  specialties  will  be  required.  As  shown  earlier,  such  a solution  is 
undesirable.  Efficient  utilization  of  medium  aptitude  personnel  will  be 
dependent  on  more  training. 

Evaluation  of  Three  Types  of  Troubleshooting  Aids 

In  this  study  (Potter  & Thomas,  1976),  two  types  of  proceduralized 
troubleshooting  aids  and  conventional  manuals  were  experimentally  compared 
for  a complex  electronic  system.  Eighteen  apprentice  technicians  and  18 
experienced  technicians  performed  actual  on-equipment  troubleshooting  prob- 
lems for  both  organizational  and  intermediate  levels  of  maintenance.  Eighteen 
new  technical  school  graduates  performed  the  same  problems  but  only  with  two 
types  of  proceduralized  data.  Previous  studies  had  indicated  that  these 
new  graduates  cannot  perform  troubleshooting  tasks  with  conventional  man- 
uals. In  addition,  eight  experienced  electronic  technicians  from  a differ- 
ent specialty  were  tested.  Job-oriented  skills  training  similar  to  that 
used  in  the  demonstration  study  by  the  basic  trainees  was  provided  to  all 
of  the  new  graduates.  This  training  was  only  2 days  in  length  and  concen- 
trated on  primary  test  equipment  use  and  JPA  use.  This  training  was  also 
administered  to  the  apprentice  and  experienced  technicians  on  an  as-needed 
basis  as  determined  by  proficiency  pretests. 

1.  Results . Proceduralized  troubleshooting  aids  were  superior 
for  all  groups  for  the  shop  level  troubleshooting  tasks  in  terms  of  time, 
spare  parts  consumed,  and  ability  to  find  the  problem.  The  new  graduates 
performed  better  with  the  more  detailed  fully  proceduralized  troubleshooting 
aid  than  with  the  less  detailed  logic  tree  proceduralized  aids.  Of  parti- 
cular interest  is  the  finding  that  the  new  graduates  performed  shop  level 
tasks  as  well  as  the  most  experienced  technicians.  The  eight  technicians 
who  were  transitioned  from  another  specialty  were  virtually  unable  to  per- 
form shop  level  tasks  with  the  conventional  manuals  and  yet  performed  as 
well  as  the  experienced  technicians  when  they  used  proceduralized  data. 

2.  Conclusions . Good  JPAs  and  Job-oriented  Training  can  support 
both  inexperienced  and  experienced  personnel.  They  can  also  be  effective 
in  shortening  the  transition  period  for  personnel  who  are  moved  to  differ- 
ent systems. 

3.  Implications.  Personnel  system  flexibility  can  be  increased 

if  proper  JPAs  can  be  developed.  Availability  of  the  JPAs  will  increase  the 
effective  manpower  available  without  further  addition  of  personnel;  i.e., 
better  utilization  will  be  obtained  from  personnel  by  decreasing  the  neces- 
sary OJT  period. 
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JPA  Utilization  Study 

An  attempt  was  made  to  determine  the  impact  of  JPAs  on  the  effec- 
tiveness of  maintenance  for  the  recently  implemented  C-141  JPAs.  Four 
bases  were  examined,  two  where  the  JPAs  had  been  introduced  and  two  where 
the  conventional  manuals  were  still  in  use.  Sample  observations  and  inter- 
views were  made  at  the  bases  and  maintenance  data  collection  records  were 
examined  for  all  bases  for  both  the  previous  periods  (pre-JPA)  and  past 
periods. 


1.  Results . No  significant  differences  were  detected  from  the 
data  collection  records  for  any  of  the  comparisons  made  (i.e.,  non- JPA 
base  vs.  JPA  base  and  pre-JPA  base  vs.  post-JPA  base,  etc.).  This  was 
believed  due  to  a number  of  factors.  First,  all  of  the  bases  were  heavily 
manned  with  experienced  personnel  during  these  periods.  The  general  obser- 
vation was  made  that  these  personnel  used  neither  the  conventional  manuals 
nor  the  JPAs.  Finally,  it  was  concluded  that  the  data  collection  records 
available  are  too  insensitive  to  the  types  of  changes  that  could  be  ex- 
pected . 


2.  Implications.  Careful  planning  must  be  made  in  future  opera- 
tional tests  of  JPA  and  Job-oriented  Training  Systems  to  ensure  that  valid 
measures  can  be  obtained. 

Conclusions  and  Recommendations 


The  present  personnel  system  has  had  a direct  impact  on  the  successful 
implementation  of  these  new  techniques  in  the  following  areas:  skill  level 

advancement  that  is  geared  to  conventional  training,  assignments  that  are 
related  to  the  award  of  a higher  skill  level,  and  promotions  that  are  tied 
to  both  skill  level  advancement  and  passing  of  a Speciality  Knowledge  Test. 
Research  data  indicates  that  at  least  some  of  the  performance  problems  ex- 
perienced by  these  graduates  can  be  overcome  by  providing  better  JPAs.  Per- 
sonnel system  changes  that  must  be  made  generally  involve  changing  the  em- 
phasis from  knowledge  to  performance  as  the  criteria  for  skill  level  advance- 
ment and  promotion. 

Even  if  personnel  system  changes  can  be  made,  the  problem  of  possible 
bias  of  the  supervisors  must  be  overcome.  It  was  suggested  earlier  that, 
in  future  efforts,  care  must  be  taken  to  ensure  that  trainees  are  permitted 
to  perform  all  of  the  tasks  required  even  though  they  cannot  pass  the  SKT. 

On  the  other  hand,  the  judgment  of  experienced  technicians  and  supervisors 
should  not  be  treated  lightly,  since  they  perform  or  supervise  others  per- 
forming these  tasks.  They  would  and  have  argued  that  a basic  understanding 
of  the  system  contributes  to  work  performance  by  enabling  the  technician 
to  actually  "solve"  problems  rather  than  fix  problems  by  rote  maintenance 
techniques.  The  new  concept  approach  is  much  more  dependent  on  many  support 
conditions  being  idealized  while  the  conventional  approach  provides  sufficient 
knowledge  to  enable  the  technician  to  circumvent  these  problems.  If  this 
is  true,  then  the  use  of  the  Job-oriented  Training/ JPA  approach  should  be 
very  restricted  in  application  rather  than  be  applied  across  the  entire 
maintenance  spectrum. 
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These  are,  of  course,  assumptions  and  should  be  subjected  to  experimental 
test.  One  approach  would  be  to  carefully  observe  many  (several  thousand) 
maintenance  actions  to  determine  the  true  incidence  rate  of  unorthodox 
problems.  These  need  not  be  restricted  to  strictly  defined  troubleshooting 
problems  but  should  address  all  tasks.  Simple  remove  and  replace  tasks 
can  quickly  become  a problem  when  replacement  parts  do  not  match  the  JPA 
illustrations. 

The  above  problem  relates  to  a more  basic  issue — namely,  does  the 
existence  of  JPAs  eliminate  the  need  for  knowledge  of  the  system?  Field 
results  thus  far  have  indicated  that  this  assumption  may  be  erroneous. 

While  we  can  surmise  that  learning  by  doing  would  be  effective  for  equip- 
ment such  as  automobiles,  this  does  not  appear  to  be  the  case  for  elec- 
tronics. Two  areas  of  research  are  suggested.  The  first  is  to  reexamine 
our  approach  to  task  analysis.  Assumptions  that  have  been  made  should  be 
evaluated  for  possible  impact  on  the  validity  of  our  approach.  Examples 
of  such  assumptions  are:  (1)  that  proper  tools,  test  equipment,  and 

trained  personnel  will  always  be  available,  (2)  that  technical  data  will 
be  available  to  the  level  required  (maintenance  concepts  frequently  change 
as  a matter  of  necessity),  (3)  that  adequate  shop  support  and  supervisory 
support  will  be  available,  and  (4)  that  adequate  time  will  be  available 
to  accomplish  the  mission.  The  second  area  of  research  is  to  determine 
if  learning  takes  place  under  the  new  concept  and  to  develop  procedures 
(if  necessary)  to  allow  it  to  take  place.  The  work  reported  by  Shriver 
and  Trexler  (1966) for  the  Coast  Guard  and  by  Post  and  Price  (1973)  for  the 
Navy  on  hybrid  aids  should  be  used  as  starting  points  in  the  electronics 
area.  Mechanical  systems  should  be  studied  as  well. 

Finally,  a common  argument  presented  is  that  the  more  experienced 
second  enlistment  technician  can  be  used  to  solve  these  unorthodox  prob- 
lems and  that  principles  training  should  be  deferred  until  this  time  so 
that  it  can  be  used  as  an  inducement  to  reenlist.  Anecdotal  evidence 
indicates,  however,  that  most  of  the  maintenance  performed  today  is, 

In  fact,  performed  by  first-term  enlistees.  A personnel  study  should  be 
made  by  survey  and  monitoring  of  data  collection  records  to  determine  the 
actual  jobs  and  percentage  of  work  that  is  now  being  performed  by  first- 
term  personnel. 
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STATUS  OF  JPA  TECHNOLOGY: 

ANALYSIS  AND  CONCLUSIONS 

Harold  R.  Booher 

Navy  Personnel  Research  and  Development  Center 


Introduction 


One  of  the  major  objectives  of  the  JPA  symposium  was  to  assess  the 
state-of-the-art  in  JPA  technology  to  determine  if  it  could  be  structured 
in  some  way  that  would  stimulate  meaningful  discourse  within  JPA  research, 
development,  and  implementation  communities.  It  was  anticipated  that  those 
areas  that  were  well  defined  might  be  highlighted  for  near-term  demonstra- 
tion in  Navy  personnel  and  training  systems  while  those  areas  not  so  well 
defined  might  be  assigned  relative  priorities  for  research.  This  objec- 
tive was  well  met  by  the  papers  presented  and  the  discussion  that  followed 
each  topic.  This  was  no  easy  task,  however,  because  of  many  of  the  mis- 
conceptions surrounding  JPAs  and — as  was  discovered  in  the  first  morning's 
meeting — there  is  no  generally  accepted  definition  of  job  performance  aid, 
even  among  those  who  have  done  considerable  research  and  development  in  the 
field. 

JPA  Technology  Base 

The  paper  presented  by  Dr.  Collins,  "Some  Perspectives  on  the  Job  Per- 
formance Aids  Technology  Base,"  outlined  a generally  broad  concept  of  JPA 
for  consideration.  His  description  of  major  conceptual  and  methodological 
developments  in  such  areas  as  motivation  theory,  job  structure,  and  learn- 
ing stimulated  discussion  well  beyond  the  textual,  highly  proceduralized 
type  of  job  aid. 

JPA  Definition 


Conceptually,  the  panel  agreed  that  job  performance  aids  can  cover 
any  information  presentation  format  and  media,  any  sensory  or  physical  un- 
burdening tool  or  activity,  and,  possibly,  the  whole  area  of  organizational 
job  design  for  matching  personnel  to  the  job.  There  is  the  basic  require- 
ment that  the  JPA  is  actually  used  at  or  near  the  job  itself.  In  the  past, 
however,  and  this  is  where  our  strong  data  base  lies,  we  have  considered 
JPAs  primarily  within  the  area  of  information  transfer.  Reid  Joyce's 
definition  appeared  quite  adequate,  therefore,  for  discussion  during  the 
rest  of  the  symposium.  He  stated  that  the  defining  characteristic  of  a 
performance  aid  is  the  capacity  to  store  information  for  later  retrieval 
in  connection  with  the  performance  of  a job.  The  aid  facilitates  perfor- 
mance by  reducing  the  memory  and  possibly  the  training  requirements  imposed 
upon  the  performer. 

JPA  Classification 

It  appears  that  a broad  classification  of  types  of  technologies  that 
form  the  JPA  technology  base  can  now  begin  to  take  shape.  The  technology 
base  can  be  broken  into  three  fairly  distinct  classes:  (1)  technology 
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associated  directly  with  job  aiding  systems,  (2)  technology  competing 
with  job  aids,  and  (3)  technology  that  supports  job  aids.  The  major  sub- 
classes that  can  be  formed  under  the  direct  job  aiding  systems  area  are 
(1)  format/content  type  of  which  the  fully  proceduralized  JPA  is  one 
basic  type,  (2)  media  (e.g.,  book,  microform,  CAI) , and  (3)  job  aid  de- 
livery system,  which  has  conventionally  been  the  technical  data  system 
for  textual  maintenance  information. 

Competing  and  Supporting  Technologies  ? 

Those  areas  that  should  be  considered  potential  tradeoffs  with 
the  JPA  (e.g.,  automatic  test  equipment,  human  engineering  maintainability 
design,  and  perhaps  job  design)  can  be  listed  under  competing  technologies. 

Supporting  technologies  are  those  that  act  to  ensure  a workable  interface 

between  a particular  type  of  JPA  and  personnel  and  training  systems.  They  i 

could  include  maintenance  simulation,  instructional  system  design,  job 
design,  and  test  and  evaluation  technology. 

The  classification  of  a technology  as  competing  or  supporting  is 
not  always  distinct.  It  depends  a great  deal  on  what  decisions  have  al- 
ready been  made  and  what  stages  of  system  development  are  being  addressed. 

Areas  of  technological  support  that  could  also  form  part  of  the  technology 
base  and  for  which  a great  deal  of  literature  is  available  are  in  instruc- 
tional system  design  for  optimum  training  strategies  and  in  job  satisfac- 
tion and  job  design.  The  work  that  has  been  done  in  these  areas  should  now 
be  reexamined  in  the  context  of  supporting  the  utilization  of  JPAs.  An 
important  technology  suggested  by  Hr.  Klesch  for  tradeoff  evaluation  with 
the  JPA  in  the  competitive  technologies  is  automatic  test  equipment.  Early 
in  the  system  concept  development,  tradeoff  studies  may  also  be  useful 
among  human  engineering  design,  automatic  test  equipment,  job  design, 
training,  and  JPA. 

Technical  Information  Transfer  Systems 

If  the  existing  technology  base  is  probed  more  deeply  within  this 
framework,  it  is  found  that  the  information  being  stored  and  transferred  is 
usually  technical  information  and,  in  particular,  technical  information  to 
be  used  by  equipment  maintenance  personnel — and  most  of  the  data  within  that 
is  for  electronics  equipment.  Here  the  technology  base  is  strong  for  two 
types  of  aids,  which  can  be  generally  summarized  under  the  "directive"  and 
the  "deductive"  type  aiding  systems.  Mr.  Post  has  described  at  least  12 
basic  format/content  types  that  can  be  built  upon  a combination  of  the 
directive  an  1 deductive  aids.  Within  any  basic  type  "format/content , " 

Dr.  Shrlver  has  formulated  five  fundamental  elements  that,  to  the  degree 
they  are  used,  will  define  the  quality  of  the  aid  and,  correspondingly, 
perhaps  the  relative  cost  of  the  aid. 

Media  of  Presentation 

The  panel's  consensus  on  the  usefulness  of  specific  media  was  that 
media  consideration  should  always  be  secondary  to  the  format/content  decision 
for  a JPA.  Specific  environmental  uses  of  audio  cassettes,  CAI,  head  devices, 
minicomputers,  and  holography  have  been  described  in  the  literature  and  form 
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part  of  the  technology  base.  Although  certain  environmental  considerations 
like  tight  quarters,  poor  illumination,  or  flight-deck  operation  might 
warrant  specific  media  for  presentation  of  the  information,  the  medium 
chosen  for  demonstration  can  show  payoffs  in  a paper  format  as  well  as 
any  other. 

Test  and  Evaluation  Methodology 

Dr.  Collins  also  points  out  that  test  and  evaluation  methodology 
itself  forms  part  of  the  technology  base.  Part  of  that  methodology  is 
measuring  those  things  that  relate  to  life-cycle  payoff.  Here  COL  Grossel 
emphasizes  the  concept  of  "Product  Improvement"  for  evaluating  payoffs. 

It  is  important  as  a measure  because  it  not  only  is  useful  on  new  systems 
that  have  good  tradeoff  opportunities,  but  on  existing  systems  as  well. 
Performance  errors,  the  traditional  measure  of  psychologists,  are  not 
particularly  useful  on  large-scale  test  and  evaluation  projects  for,  as 
Dr.  Shriver  noted,  "errors  don't  translate  into  money  (or)  . . . produc- 
tivity." 

Adequacy  of  JPA  Technology  Base 

The  panel  recognized  that  the  positive  assessments  of  JPA  experi- 
mental evaluations  made  by  Rowen  (1973),  Shriver  (1975),  and  King  (1975) 
must  be  qualified  with  the  knowledge  that  all  of  those  tests  were  short- 
term and  did  not  consider  the  full  implications  of  long-term  applications. 

For  the  most  part,  the  answer  must  be  no  to  Dr.  Collins'  question,  "Does 
the  technology  base  adequately  address  the  integration  of  the  range  of 
psychological  factors  in  the  individual,  job,  and  the  environment?"  A 
final  comment  regarding  the  adequacy  of  the  JPA  technology  base  was  the 
consensus  that,  although  a number  of  demonstrations  have  been  done  over 
the  past,  the  basic  and  exploratory  research  support  has  been  quite  weak. 

In  Mr.  Post's  words: 

We  have  had  no  underlying  technology  base  development. 

In  the  50' s what  was  being  done,  about  JPAs  that  we  can 

draw  upon?  Hardly  anything.  In  the  60's  what  was  done? 

Most  of  the  inputs  for  the  whole  program  came  from  the 

Air  Force  PIMO  Program. 

It  is  for  that  reason  that  the  present  NAVPERSRANDCF.N  project  has  some 
good  but  quite  limited  JPA  candidates  for  integration  within  the  per- 
sonnel and  training  communities.  6.1  and  6.2  R&D  efforts  should  probably 
by  initiated  now  to  fill  gaps  in  the  technology  base  and  to  explore  the 
feasibility  of  some  truly  new  concepts  for  JPAs. 

Optimum  Job  Performance  Aid 

A problem  facing  any  program  manager  of  a new  weapon  system  is  the 
selection  of  the  best  JPA  technique  for  his  particular  system,  considering 
specific  operational  constraints,  manning  skills  and  levels,  and  availability 
of  funds.  In  the  past  within  the  technical  manual  system,  for  example,  he 
has  had  the  choice  of  almost  100  acronyms  to  select  from  (SIMM,  FOMM,  MDC, 
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FPJPA,  MIARS,  etc.),  but  little  idea  of  relative  advantages  or  disadvantages 
of  each.  It  would  be  highly  desirable,  therefore,  for  the  PM  to  have  a guide 
or  an  algorithm  for  selecting  job  aid  techniques  that  are  optimum  for  his 
specific  requirements. 

Directive  and  Deductive  JPAs 


Mr.  Post's  paper,  "Selection  of  Formats  and  Media  for  Presenting 
Maintenance  Information,"  showed  an  intriguing  approach  being  considered 
by  the  NTIPP  for  matching  systems  to  presentation  techniques.  Inherent 
in  the  methodology,  of  course.  Is  the  hypothesis  that  job  aid  utilization 
levels  will  be  much  higher  than  the  utilization  typically  shown  by  the 
conventional  technical  manual  (less  than  5%  of  the  total  maintenance  time). 

The  approach  proposed  by  Post  takes  advantage  of  the  available  tech- 
nology base  where  it  is  strongest  (i.e.,  maintenance  technical  data  presen- 
tation) and  makes  use  of  desirable/undesirable  aiding  characteristics  that 
tend  to  be  classifiable  under  either  "directive"  or  "deductive"  types  of 
format/content.  Examples  of  the  directive  aid  are  the  FPJPA  or  the  main- 
tenance action  procedures  in  a technical  manual.  Examples  of  the  deductive 
aid  may  be  MDCs,  functional  flow  diagrams,  or  electrical  schematics.  It  is 
possible,  therefore,  with  the  framework  set  forth  in  the  JPA  technology  base, 
to  catalog  all  of  the  job  aiding  systems  with  acronyms  into  a much  smaller, 
more  meaningful  number  of  basic  aiding  systems. 

Fully  Proceduralized  JPA 

As  discussion  progressed  on  this  topic,  another  piece  of  the  tech- 
nology base  became  more  evident.  It  seems  that  whenever  the  task  of  being 
aided,  whether  operator  or  maintenance,  can  be  classified  as  nontrouble- 
shooting, the  fully  proceduralized  aid  improves  performance,  no  matter  whether 
the  person  being  aided  is  experienced  or  inexperienced.  For  the  nontrouble- 
shooting conditions,  then,  the  selection  problem  reduces  to  a cost  tradeoff 
on  how  much  of  the  FPJPA  can  be  afforded.  Here,  Shriver's  fundamental  ele- 
ments, which  in  the  nontroubleshooting  case  reduce  to  four,  can  be  used  to 
assess  how  much  quality  the  PM  will  be  willing  to  pay  for  in  return  for 
training  and  personnel  cost  savings. 

The  selection  mode]  proposed  by  Mr.  Post  would  be  most  useful  in 
troubleshooting  situations,  but  it  is  in  the  troubleshooting  areas  that  the 
tradeoff  data  for  showing  relative  superiority  for  one  aid  over  the  other 
are  weakest.  A fully  proceduralized  troubleshooting  format  does  not  appear 
feasible  for  all  troubleshooting  cases,  even  if  it  could  be  afforded.  For 
long  checkout  chains,  Joyce  and  Shriver  stated  that  it  is  an  inherently 
inefficient  (i.e.,  time  consuming)  procedure  and,  like  automatic  test  equip- 
ment, will  not  be  totally  reliable  if  faults  occur  that  were  not  considered 
originally. 

Performance  Data  for  Selection  Criteria 


The  panel  was  in  agreement  that  the  ability  of  a PM  to  select  one 
troubleshooting  aid  over  another  or,  better,  yet,  to  design  an  optimum  aid 
to  meet  his  system  and  operational  constraints,  manning  levels,  personnel 
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skillB  and  costs  would  be  highly  desirable.  But,  as  Post  emphasized,  from 
his  past  discussion  with  program  managers,  "They  are  not  going  out  on  a limb, 
they  want  concrete  data  and  relatively  good  assurances."  This  was  reiterated 
by  Dr.  Blanchard  who  felt  that  the  only  way  to  make  the  system  work  is  to 
bring  the  decision  maker  some  kind  of  established  set  of  payoff  expectan- 
cies. It  became  apparent  throughout  this  discussion  that  sufficient  per- 
formance data  for  selecting  one  troubleshooting  technique  over  another  for 
developing  a useful  program  manager's  guide  does  not  exist  at  this  time  and 
probably  should  be  addressed  as  a relatively  high  priority  of  the  NTIPP. 

JPA  Costs 


! 


Equally  important,  however,  if  not  more  so,  is  the  problem  of  rela- 
tive costs  of  aids.  Dependable  data  on  the  costs  of  JPAs  are  almost  non- 
existent. There  was  virtually  no  agreement  among  the  panel  members  on  this 
matter.  Estimates  of  the  FPJPA  compared  to  conventional  technical  manuals 
ranged  from  little  difference  to  five  to  ten  times  greater.  As  one  member 
pointed  out,  there  is  really  no  basis  for  comparing  one  format  versus  another 
if  content  and  quality  are  not  quantified.  For  example,  a conventional  TM 
could  cost  more  if  all  the  technical  data  information  other  than  maintenance 
action  information  is  included  in  the  cost.  On  the  other  hand,  if  a detailed 
level  of  analysis  is  required  for  the  FPJPA  and  this  is  compared  against 
gross  maintenance  procedures  in  TMs  where  the  writer  assumed  the  reader  to 
be  highly  experienced  or  trained  in  maintenance  procedures,  the  FPJPA  would 
cost  considerably  more. 

Dynamic  Aiding  Characteristics 

Whatever  the  aid  selected,  it  is  also  important,  as  Post  suggests, 
to  consider  the  dynamic  features  of  the  aid,  not  only  as  to  its  ability 
to  be  revised  by  user  feedback  or  to  conform  with  system  changes,  but  also 
to  progress  with  advances  in  the  personnel  career  ladder.  Some  of  the  work 
already  done  on  a hybrid  aid  by  NAVAIR  (.combination  of  directive  and  deduc- 
tive, with  overlapping  information  so  the  maintainer  could  gradually  progress 
from  a highly  directive  aid  early  in  his  career  to  a highly  deductive  aid 
later  in  this  career)  would  find  application  within  this  selection  model  and 
possibly  further  reduce  the  requirement  to  choose  between  one  or  the  other. 


JPA  Integration  Model 


Dr.  Collins  likened  the  JPA  selection  problem  to  that  of  the  human 
factors  engineering  integration  project  in  the  conceptual  design  process. 
There  you  may  be  dealing  with  data  and  principles  that  provide  guidelines 
for  making  decisions  about  whether  it's  going  to  be  a man  in  the  system  or 
a piece  of  hardware  in  the  system.  With  respect  to  the  JPA,  this  brings 
up  the  question  of  where  in  the  process  JPAs  ought  to  be  considered  and  what 
decisions  should  be  considered  at  each  stage.  The  selection  model  should 
be  sophisticated  enough  to  at  least  tell  what  the  tradeoffs  are  with  train- 
ing or  manning  skill  levels.  If  some  of  the  performance  data  and  cost  data 
gaps  can  be  filled,  perhaps  a model  like  the  one  started  by  NAVAIR  with 
N.  C.  State  (Ayoub,  Smillie,  Edsall,  & Muller,  1976)  could  be  used  for 
addressing  the  relative  advantages  and  disadvantages  of  one  aid  versus 
another,  or  JPA  versus  training.  It  became  quite  clear  by  this  point, 
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however,  that  the  problem  of  JPA  utilization  in  training  and  personnel  design 
is  going  to  be  much  greater  than  selecting  the  optimum  aid  from  a host  of 
candidates . 


JPA  Delivery  System 

One  of  the  most  fundamental  assumptions  of  using  job  performance  aids 
to  reduce  training  requirements  or  to  accomplish  a job  with  lower  skill  per- 
sonnel is  dependability  of  the  aid.  It  must  be  accurate  and  available  at 
the  time  the  job  is  being  performed.  To  the  degree  it  is  not,  additional 
training  or  backup  personnel  must  be  made  available.  When  the  JPA  is  to 
display  operator  or  maintenance  information,  the  available  engineering  data 
base  is  the  starting  point  for  its  development.  From  this,  both  on-the-job 
information  (operator  and  maintainer)  and  training  information  can  be  de- 
veloped and  placed  into  textual  formats  or  special  media  for  presentation. 

Independence  of  Training  and  Technical  Data  Systems 

In  the  past,  the  technical  manual  people  have  been  the  main  users 
of  on-the-job  operator  and  maintenance  information,  but  have  assumed  the 
personnel  using  the  technical  manual  will  have  had  considerable  training  be- 
fore using  the  manual.  The  training  people,  on  the  other  hand,  may  use  the 
same  engineering  data  base  to  develop  learning  materials  aimed  primarily 
at  Improving  those  knowledges  and  skills  that  can  be  retained  in  memory. 

The  result  of  these  two  almost  independent  systems  is  a type  of  redundancy 
that  tends  to  take  care  of  a great  deal  of  uncertainty  that  exists  in  the 
operational  world.  The  adverse  effects  of  inaccurate  and  unavailable  tech- 
nical data  is  offset  (and,  to  a great  extent,  disguised)  by  the  availability 
of  highly  trained  personnel.  Conversely,  the  availability  of  a good  main- 
tenance manual  in  the  hands  of  an  experienced  maintenance  technician  can 
offset  (and  similarly  disguise)  the  effects  of  inadequately  trained  per- 
sonnel . 

The  fact  that  the  two  systems  are  independent  (at  least  in  the  Navy) 
was  expressed  most  succinctly  by  Post,  "Articulation  between  the  training 
process,  the  process  by  which  training  materials  are  developed,  and  the 
process  by  which  JPAs  or  technical  manuals  are  developed  does  not  exist." 

In  short,  training  people  and  technical  manual  people  do  not  speak  and  do 
not  attempt  to  speak  the  same  language. 

There  are  three  major  problems  with  the  operation  of  two  relatively 
independent  systems  In  this  way:  (1)  There  is  no  assurance  that  either 

system  is  working  sufficiently  well  to  compensate  for  weaknesses  in  the 
other — thus  allowing  tremendous  operational  deficiencies,  (2)  in  the  best 
of  systems,  the  dual  support  systems  approach  is  extremely  costly,  and  (3) 
the  masking  effects  of  one  system  on  the  other  make  it  almost  impossible  to 
pinpoint  major  faults  so  that  intelligent  decisions  can  be  made  on  where 
to  cut  costs  and  where  to  improve  efficiency. 

Future  JPA  Delivery  System  Alternatives 

The  move  toward  the  JPA  to  cut  costs  in  training  and  personnel  has 
to  change  the  way  of  procuring  and  producing  technical  data  for  the  two 
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redundant  systems.  Assuming  a methodology  for  selecting  an  optimum  aid  can 
be  developed  for  predicting  payoff  expectancies  relative  to  training  and 
manning,  the  next  problem  to  be  solved  becomes  essentially  a policy  decision 
by  top  Navy  management.  This  question  centers  around  what  delivery  system 
will  be  used  to  produce,  distribute,  and  update  JPAs.  Conference  partici- 
pants suggested  three  alternatives:  (1)  modify  the  existing  technical  data 

system  to  include  training  requirements,  such  as  being  attempted  by  the 
NTIPP  program,  (2)  create  a whole  new  system  for  delivery  and  update — pos- 
sibly within  the  training  command,  and  (3)  combine  training  and  technical 
data  into  one  system,  following  the  example  being  set  by  the  Army  in  their 
ITDT  program.  The  first  of  these  is  probably  the  best  short-term  goal  for 
the  Navy,  but  the  third  suggestion,  to  combine  the  two  major  areas  of  train- 
ing and  technical  data  so  that  both  are  speaking  the  same,  albeit  a new 
language,  appears  to  be  the  more  advanced,  most  efficient  long-term  solu- 
tion. 

Integrated  Training  and  Technical  Data 

If  the  provincialism  of  Navy  training  and  Navy  technical  data  systems 
could  be  overcome,  tremendous  savings  in  data  and  training  costs  could  be- 
come a possibility.  As  Dr.  Shriver  suggests: 

The  pattern  should  be  to  stop  talking  about  such 
things  as,  this  is  training,  this  is  documentation,  and 
this  is  training  aids,  but  that  it  is  all  one  thing  . . . 
the  old  assumption  was  that  training  is  supposed  to  de- 
scribe the  equipment  . . . Bring  the  two  together  and 
the  individual  will  figure  out  what  to  do:  A new  approach 

is  to  analyze  the  whole  thing  . . . figure  out  what  he  is 
supposed  to  do,  and  tell  him. 

When  one  hears  the  arguments  for  technical  data  being  the  prime  mover  (e.g., 
Mr.  Johnson's  emphasis  on  needs  for  configuration  control  and  close  associa- 
tion to  engineering  drawings)  versus  that  of  training  (e.g.,  Mr.  Post's 
comments  on  user  feedback  and  Dr.  Shriver 's  statements  about  training  command 
assurance  of  validation),  it  becomes  even  more  evident  that  both  have  some- 
thing to  offer  but  need  to  be  addressing  a common  goal. 

Technology  Limitations  in  JPA  Accuracy 

Mr.  Johnson  brought  up  points  in  his  paper,  "Problems  in  Procuring, 
Producing,  and  Evaluating  JPAs,"  that,  for  the  most  part,  probably  will  not 
go  away  completely  with  any  delivery  process.  There  seems  to  be  a limit  to 
how  much  accuracy  can  be  incorporated  into  JPAs.  His  information  on  the 
two  major  sources  of  error  (i.e.,  human  limitations  in  predicting  all  pos- 
sible faults  for  all  systems  and  equipment  and  errors  due  to  hardware  re- 
visions) illustrates  why  it  is  not  reasonable  to  expect  error-free  documen- 
tation in  complex  systems.  This  knowledge,  coupled  with  the  first-hand 
experiences  of  almost  everyone  at  the  conference  of  never  having  had  totally 
error-free  JPAs  to  work  with,  makes  it  only  too  clear  that  there  is  a 
technological  limitation  to  how  much  accuracy  is  obtainable  with  JPAs. 

This  confirms  earlier  suspicions  that  JPAs  cannot  (at  least  in  the  trouble- 
shooting areas)  totally  eliminate  the  need  for  training  of  skills  greater 


than  those  that  can  be  imparted  totally  by  the  JPA  itself.  The  panel  was 
also  sure,  however,  that  the  level  of  quality  could  be  made  quite  high, 
provided  maximum  efforts  are  made  toward  tight  enforcement,  use  of  pro- 
fessional writers,  and  a training  validation. 

User  Acceptance 

Whenever  a new  piece  of  equipment,  a new  system,  or  a new  way  of  doing 
business  is  introduced,  it  must  first  gain  user  acceptance  before  its  value 
can  be  realized.  The  paper  by  Reid  Joyce,  "User  Problems  in  JPA  Utiliza- 
tion," provided  an  excellent  status  report  on  the  general  nonacceptance 
of  JPAs  even  though  almost  every  experimental  evaluation  has  shown  that 
technicians,  both  experienced  and  inexperienced,  make  fewer  errors  on  the 
job  when  using  a JPA.  He  emphasized  that  much,  if  not  most,  of  the  problem 
lies  in  the  fact  that  there  are  a large  number  of  users,  in  addition  to  the 
person  doing  the  job,  who  are  affected  by  JPAs  being  introduced  into  the 
system.  These  include  the  technicians'  peers  and  supervisors,  nonmainten- 
ance users  of  the  documentation  system,  and  all  those  connected  with  hard- 
ware and  logistics  problems  at  the  program  manager  level. 

"Real"  and  "Imaginary"  User  Acceptance  Problems 

For  the  purpose  of  organizing  the  technology  base,  it  is  useful  to 
treat  the  two  types  of  user  acceptance  problems  separately.  One  type  is 
what  may  be  considered  "real"  problems  with  user  acceptance  because  they 
have  to  do  with  acceptance  or  rejection  of  the  aid  itself  by  the  field  user 
himself  because  of  his  initial  or  acquired  attitude  toward  how  the  aid  effects 
his  actual  performance  on  the  job.  The  other  type  may  be  considered  to  be 
"imaginary"  with  respect  to  those  kinds  of  things  that  may  be  inherent  in  the 
aid,  but  that  nevertheless  can  override  acceptance  of  the  aid.  These  are 
problems  of  general  acceptance  by  the  operational  and  support  systems  already 
in  existence.  The  "imaginary”  problems,  of  course,  must  be  dealt  with  for 
successful  long-term  demonstration  or  eventual  implementation,  but  they 
should  not  be  confused  with  the  inherent  advantages  or  limitations  of  the 
job-aiding  technology  itself. 

The  first  set  of  problems,  those  which  contribute  directly  to  aid 
acceptance,  include  things  like  the  actual  limitations  of  the  JPA  (e.g., 
inaccuracies,  insufficient  level  of  detail,  or  too  much  detail)  but  they 
also  include  a lot  of  attitudinal  characteristics  of  the  technician.  For 
example,  as  Joyce  warns,  if  the  JPA  is  introduced  as  something  to  help 
"dummies"  or  something  to  make  the  job  "simpler"  rather  than  "easier,"  it 
will  be  rejected.  Also,  no  matter  how  good  the  aid  is,  it  must  compete, 
as  Post  suggests,  with  other  styles  of  operation:  "In  the  eyes  of  the  user, 
is  the  JPA  better,  worse,  or  the  same  as  getting  advice  from  peers,  working 
it  out  according  to  their  own  ingenuity,  or  admitting  to  their  supervisor 
they  don't  know  and  ask  for  advice?" 

JPAs'  Contribution  to  Job  Satisfaction 

The  data  available  support  the  concept  of  JPAs  as  a means  of  en- 
hancing job  satisfaction  that  should  provide  a strong  motivation  for  user 
acceptance.  For  example,  there  are  sufficient  learning  concepts  to  infer 
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that  job  satisfaction/motivation  can  improve  through  satisfactory  on-job 
performance.  Dr.  Collins  states  in  this  paper:  "Learning,  change,  and 

growth  are  best  facilitated  by  an  integrated  process  involving  immediate 
experience,  validation  of  that  experience,  modification  of  this  behavior, 
and  the  choice  of  new  experiences."  Also  attitude  information  that  has 
been  collected,  much  of  it  by  panel  members  themselves  (e.g..  Post,  Shriver, 
& Joyce)  has  indicated  a strong  preference  for  JPAs  over  conventional 
materials.  Even  the  personnel,  reported  in  Klesch's  paper,  could  be  assumed 
to  have  had  high  morale  and  good  attitudes  toward  the  JPA  initially. 

Adequacy  of  User  Acceptance  Technology  Base 

Although  the  first  set  of  problems  is  substantial,  the  technology 
base  seems  to  be  on  firmer  ground  there  than  with  the  second  set.  The  con- 
ference participants  were  confident  that  many  of  the  direct  user  problems 
had  been  addressed  in  the  past  and  could  be  solved  with  what  we  now  have 
available.  The  second  set  of  problems,  gaining  acceptance  of  all  the  in- 
direct users,  is  the  challenge  which  remains  to  be  solved  by  the  JPA  com- 
munity. 

JPA/Training  Interface 

Dr.  Shriver's  paper,  "New  Directions  for  Information  Transfer  Research 
in  Maintenance  Jobs,"  reiterates  the  basis  for  the  NAVPERSRANDCEN  project. 
There  appear  to  be  major  potential  payoffs  on  the  order  of  100  to  1 in  per- 
sonnel and  training  maintenance  costs  with  the  utilization  of  JPAs  even  if 
the  JPAs  cost  20  percent  to  50  percent  more  to  procure  than  conventional 
technical  maintenance  action  materials.  He  notes  that  studies  over  the 
past  20  years  show  reductions  in  training  on  the  order  of  50  percent  to 
90  percent  with  corresponding  increases  in  job  productivity  as  high  as  50 
percent  and  60  percent.  These  gains  cannot  be  realized,  however,  by  the 
simple  introduction  of  the  JPA  into  the  current  maintenance  and  training 
system. 

Adequacy  of  Interface  Technology 

The  questions  raised  by  Merle  Malehorn  In  his  paper  "Impact  of 
Job  Performance  Aids  in  Personnel  and  Training"  (In  Rowan,  1975)  on  career 
patterns  for  personnel,  cross-training,  backup  training,  on-the-job  train- 
ing, generalized  skills,  and  measures  of  proficiency  are  still  valid.  The 
JPA/training  interface  becomes  very  critical  to  the  actual  achievement  of 
any  of  the  potential  JPA  gains.  As  already  discussed,  the  technology  base 
for  JPA  development  is  excellent  for  nontroubleshooting  tasks  for  operators 
and  maintainers  and  offers  several  strong  contenders  for  troubleshooting 
tasks.  Also,  the  research  base,  in  education  and  training  on  instructional 
system  design,  appears  quite  strong  for  determining  optimum  strategies  in 
achieving  basic  iearning  objectives.  The  place  where  the  technology  base 
is  weak  is  at  the  interface  of  these  two  technologies.  It  would  probably 
be  helpful  at  this  point  to  consider  some  of  the  concepts  and  assumptions 
offered  by  JPA  specialists  for  dealing  with  the  questions  raised  by 
Mr.  Malehorn. 


Common  Task  Analysis 


One  of  the  most  fundamental  assumptions  of  JPA  proponents  is  that 
a common  task  analysis  be  conducted  for  both  training  and  JPA  development. 
Four  of  Shriver's  fundamental  elements  deal  with  the  different  levels  of 
analysis  required  for  both  training  and  performance  aids.  One  of  the 
earliest  decisions  would  be  for  the  analyst  to  determine  which  items  should 
go  into  the  book  (JPA)  and  which  into  the  head  (training).  The  questionable 
area,  here  again,  is  in  troubleshooting.  As  Mr.  Johnson  pointed  out,  there 
is  no  standard  or  optimum  means  of  developing  troubleshooting.  There  is  a 
paragraph  in  FPJPA  specifications  that  says,  "find  an  experienced  tech- 
nician to  do  the  troubleshooting." 

There  are  some  assumptions  here  for  training.  One  of  these  is  that 
the  primary  strategy  (in  electrical,  electronic,  electromechanical,  and 
hydraulic  maintenance)  to  be  learned  is  signal  tracking.  The  way  to  opti- 
mize this  is  thought  to  be  through  emphasis  on  understanding  functional 
relationships  of  major  components  at  a block  diagram  level  for  specific 
systems  as  opposed  to  learning  general  circuit  theory  or  fluid  flow 
dynamics.  An  interface  question  to  be  answered  however,  for  any  specific 
system,  is  how  much  of  this  could  also  go  into  a JPA  rather  than  in  a 
classroom. 

Career  Patterns 


A new  concept  is  beginning  to  take  place  in  an  attempt  to  help  inte- 
grated JPAs  and  training  handle  career  patterns.  Post  suggests  we  stop 
talking  about  experienced  and  inexperienced  personnel  and  recognize  that 
there  are  gradations  of  experience.  The  job  aids  and  the  training  in  com- 
bination must  not  only  get  productive  labor  at  the  lowest  level  but  also  has 
to  prepare  the  technician  for  progression  to  the  next  plateau  of  his  career. 
To  the  degree  this  is  attempted  through  the  aid  rather  than  formal  training, 
the  hybrid  aid  concept  developed  by  NAVAIR  about  4 years  ago  should  be  con- 
sidered a plausible  candidate. 

One  of  the  oldest  assumptions  of  the  JPA  community  was  set  forth 
by  Dr.  Shriver;  that  is,  to  put  the  first-term  enlistees  to  work  immedi- 
ately, and,  with  the  JPA,  some  productive  work  can  be  achieved  while  he  is 
learning  higher  skills  on  the  job.  If  the  technician  signs  up  for  another 
tour,  he  can  then  be  given  formal  theory  training.  This  person  with  both 
theory  training  and  several  years  of  on-job  experience  can  become  part  of 
the  backup  used  to  solve  the  difficult  troubleshooting  problems  that,  for 
one  reason  or  another,  were  not  covered  adequately  by  the  job  aid.  A special 
path  for  the  high  aptitude  first-tour  enlistee  should  also  be  available 
in  such  a system. 

Generalized  Skills 


Another  important  aspect  of  career  patterns  through  integrated  train- 
ing and  job  aiding  to  be  considered  is  the  development  of  generalized  skills. 
Here  again,  it  is  primarily  troubleshooting  where  this  is  a concern.  A good 
maintenance  technician  should  be  able  to  carry  more  than  basic  skills  learned 
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on  one  type  of  equipment  to  other  similar  types.  Because  of  many  of  the 
problems  discussed  on  the  limitations  in  the  technology  for  delivering  a 
totally  reliable  JPA  and  the  realistic  view  that  operational  conditions 
seldom  if  ever  meet  the  "idealized"  planned  conditions,  career  maintenance 
personnel  must  learn  to  solve  problems  as  well  as  to  fix  them  by  rote.  The 
hybrid  aid  concept  may  be  useful  here  too  for  achieving  generalizable  cogni- 
tive skills  when  used  in  conjunction  with  on-job  training.  As  Joyce  states 
in  his  paper,  "The  ultimate  solution  will  probably  include  JPAs  used  by 
inexpensively-trained,  early  enlistment  people,  and  more  conventional  aids 
used  by  more  experienced  people  who  have  received  supplemental  training  along 
the  way  or  at  the  beginning  of  their  next  enlistment."  Job  design  technology 
would  need  to  be  used  to  match  a large,  less  complex  workload  for  the  large 
number  of  JPA  users  and  a smaller,  but  critical,  backup  workload  for  the 
conventional  aid  user. 

Measure  of  Proficiency 

Finally,  with  respect  to  the  new  JPA/ training  interface,  and  personnel 
career  patterns,  it  is  absolutely  essential  that  measures  of  proficiency  be 
designed  around  actual  job  performance  skills  rather  than  conventional  know- 
ledge tasks  as  basic  criteria  for  skill  level  advancement,  job  assignment, 
and  promotion. 

Effects  on  Personnel  System 

It  was  with  the  presentation  of  John  Klesch's  paper  that  the  full  mag- 
nitude of  the  problem  of  implementing  a new  technology  like  the  JPA  came 
clearly  into  focus.  Even  though  the  job  aid  technology  base  is  sound,  the 
optimum  aiding  technique  is  selected,  the  delivery  system  is  providing  an 
accurate  and  timely  JPA,  and  the  field  user  has  accepted  and  is  using  the 
JPA,  the  JPA  is  doomed  to  failure  if  long-term  personnel  effects  are  not 
considered.  Again  those  questions  raised  by  Merle  Malehorn  at  the  confer- 
ence on  "Improved  Information  Aids  for  Technicians"  in  1975  are  those  to 
be  answered.  In  summary,  they  include  personnel  considerations  like  job 
structures,  assignment  structures  and  processes,  processes  for  personnel 
classification,  advancement  procedures,  and  recruiting  and  retention. 

Test  Subject  Careers 

No  JPA  experimental  demonstration  has  really  attempted  to  deal 
with  long-term  personnel  effects  in  the  past,  but  they  must  be  primary 
considerations  in  any  future  research.  In  fact,  with  the  knowledge  of 
the  damaging  long-term  effects  on  the  personnel  trained  with  JPAs  reported 
in  Klesch's  paper,  it  would  be  irresponsible,  if  not  raising  legal  liability, 
to  do  further  JPA  experimentation  without  providing  satisfactory  career  paths 
for  test  subjects.  It  is  useful  here,  as  with  the  User  Acceptance  Technology 
discussion,  to  separate  those  things  that  may  adversely  effect  the  techni- 
cian's career  because  of  inherent  limitations  in  the  JPA  concept  itself 
from  those  that  are  almost  entirely  related  to  "the  way  of  doing  business." 

Personnel  System  Cooperation 

Assuming  the  training  interface,  the  assignments,  and  criteria  for 
prof!  iency  are  fully  considered  through  an  integrated  approach  from  the 
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available  technology  base  in  JPA,  training,  and  job  design,  most  of  the 
questions  raised  in  the  personnel  system  can  be  classified  as  system  imple- 
mentation problems.  And  the  results  of  the  NAVPERSRANDCEN  demonstration 
will,  of  course,  be  needed  to  justify  any  major  changes  in  the  personnel 
system.  Until  the  NAVPERSRANDCEN  demonstration  is  completed,  however,  it 
will  not  be  known  whether  JPAs  can  positively  influence  personnel  careers  and 
upgrade  skills.  It  is  known  that  JPAs  simply  introduced  into  existing  logis- 
tics support  systems  without  corresponding  system  changes  will  have  a nega- 
tive effect  on  personnel  careers.  In  order  to  conduct  a demonstration, 
therefore,  special  case  procedures  for  the  test  subjects  must  be  developed 
and  approved  at  the  BUPERS  level  for  classification,  assignment  and  advance- 
ment. In  addition,  at  the  field  level,  the  confounding  effects  of  peer  re- 
jection and  supervisor  bias  must  be  rigorously  controlled. 

Recruitment  and  Retention 

Recruitment  and  retention  aspects  of  this  personnel  system  are  dif- 
ferent. They  raise  special  considerations  that  relate  more  directly  to  con- 
cepts of  the  JPA  itself.  For  example,  Klesch  brings  up  the  possibility 
that  many  first-term  enlistees  are  already  performing  useful  productive 
labor.  His  impression  is  that,  in  the  Air  Force,  many  of  the  first-term 
airmen  are  indeed  performing  a great  deal  of  maintenance.  If  that  is 
happening,  it  certainly  is  not  being  done  on  any  systematic  basis.  But, 
to  the  degree  that  special  groups  can  be  identified  that  do  make  good  use 
of  new  enlistees,  they  may  provide  clues  for  job  design  and  might  provide 
useful  information  for  the  concept  that  increased  productivity  can  lead 
to  greater  job  satisfaction  and,  consequently,  increased  retention.  A 
problem  mentioned  in  the  recruitment  area  with  respect  to  JPA  is  also 
critical.  There  could  be  fewer  first-term  enlistees,  if  the  incentive 
"Navy  training"  provided  in  the  past  is  removed.  Surveys  should  probably 
be  conducted  in  the  near  term  to  determine  how  much  "truth"  there  is  to 
these  two  JPA  personnel  system  effects. 


Implications  for  JPA  Demonstration  and  Implementation 

Many  specific  recommendations  for  JPA  research  and  development  are  given 
in  the  papers  and  in  the  selected  verbatim  comments  in  the  appendix.  They 
all  have  implications,  either  for  the  current  NAVPERSRANDCEN  test  and  evalu- 
ation project  or  for  the  long-term  implementation  of  JPA  technology  into 
Navy  weapon  logistics  support  systems.  Some  of  the  recommendations  appear 
most  directly  applicable  to  the  NAVPERSRANDCEN  project  goals;  others  appear 
outside  the  scope  of  the  immediate  project,  but  relate  to  it  indirectly  if 
the  payoffs  anticipated  in  the  demonstration  project  are  ever  to  be  realized 
in  the  operational  world. 

Long-term  Demonstration  Recommendations 

A general  conclusion  can  be  drawn  that  directly  influences  the  scope 
and  direction  of  the  NAVPERSRANDCEN  advanced  development  performance  aids 
project.  The  panel  concurred  that  the  JPA  concepts  and  approaches  in  the 
available  technology  base  are  sufficiently  broad  and  flexible  to  show  major 
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payoffs  only  within  maintenance  areas  using  technical  information  presenta- 
tion for  troubleshooting  and  nontroubleshooting  tasks.  The  data  are  strong 
here  and  should  be  pursued  with  a longitudinal  demonstration  (1-5  years) 
for  effects  on  training  and  personnel.  The  demonstration  should  show  the 
ability  to  influence  costs  across  the  system  life  cycle  and  to  positively 
influence  personnel  careers  and  upgrade  skills. 

The  test  cases  selected  for  demonstration  should  probably  agree 
wherever  possible  with  those  suggested  by  Klesch.  He  suggests  a complex 
task  electronics  system,  a not-so-complex  electronic  system,  a mechanical 
system,  and  a system  that  integrates  a mechanical  system  with  a complex 
digital  electronics  system.  For  at  least  some  of  the  test  cases  chosen, 
nontroubleshooting  tasks  should  be  covered.  These  could  cover  both  opera- 
tor and  maintainer  tasks  equally  well.  The  type  of  format/content  JPA  for 
nontroubleshooting  should  be  the  FPJPA. 

Each  test  bed  chosen  should  probably  address  troubleshooting  tasks. 

The  selection  methodology  being  developed  on  the  NTIPP  program  should  be 
helpful  to  NAVPERSRANDCEN  in  the  program  manager's  guidelines  to  be  developed. 
However,  experimental  performance  data  and  cost  data  for  comparing  one  con- 
cept versus  another  for  use  in  a payoff  model  will  be  required  before  it 
can  be  used  reliably  by  a program  manager.  The  troubleshooting  format/con- 
tent should  probably  be  a combination  of  "directive"  and  "deductive"  aiding 
selected  from  techniques  known  in  FPJPA,  FOMM,  work  package  concept,  and 
logic  tree  block  diagrams.  The  NAVAIR  hybrid  aid  or  similar  concept  should 
be  reviewed  for  applicability  to  each  troubleshooting  test  case. 

Methodological  Study  Recommendations 

Cost-effective  use  of  training  and  training  systems  in  conjunction 
with  JPA  is  a most  critical  factor.  A near-term  study  should  address  how 
much  "deductive"  troubleshooting  should  be  left  to  the  JPA  and  how  much 
should  be  part  of  a job-based  training  package.  Methodological  problems 
that  should  have  highest  priority  include  (1)  methods  of  integrating  job- 
based  training  with  JPA  technology,  (2)  a study  of  the  implications  of 
JPA  technology  for  job  design  on  personnel  careers  and  job  satisfaction, 
and  (3)  the  development  of  a JPA-based  personnel  system  concept. 

The  personnel  system  concept  would  define  a limited  model  integrated 
across  training,  personnel,  and  job  design  with  the  objective  of  achieving 
full  utilization  of  JPA  technology  wherein  an  operator  or  maintainer  could  be 
given  work  assignments  early  in  his  career.  The  concept  should  also  pro- 
vide a system  by  which  career-oriented  personnel  could  seek  preparation 
and  experience  for  advancement.  Such  factors  as  career  patterns,  channels 
of  advancement,  proficiency  assessment,  motivation  and  morale,  career-oriented 
training,  and  accelerated  OJT  would  need  to  be  considered.  The  personnel 
system  concept  would  provide  a step  toward  the  broader  problem,  that  of 
integrating  the  JPA/training/ job  design  tradeoffs  into  the  total  weapon 
system  and  logistics  support  research  and  development  stages. 


JPA  Technology  Implementation  Recommendations 


The  conference  participants  agreed  that  a total  model  should  start 
by  laying  out  the  communication  interfaces  with  training,  manning,  technical 
documentation,  and  the  other  logistics  support  areas.  Ideally,  at  each  de- 
cision stage  in  the  conceptualization,  design,  and  development  of  each  major 
weapon  system,  the  PM  could  exercise  decisions  with  quantitative  criteria 
and  reliable  tradeoff  methodology  among  the  human  resources  and  the  other 
logistic  support  areas.  The  Human  Factors  Engineering  Integration  Project 
of  NSRCD,  Annapolis,  should  be  reviewed  for  applicability  to  the  larger 
problem  since  it  addresses  problems  that  may  parallel  those  of  the  JPA 
technology  implementation.  The  recent  work  of  Essex  Corporation  (Collins, 
1977)  for  NAVPERSRANDCEN  on  assessing  strategies  for  mapping  the  processes 
and  defining  the  communications  networks  that  interface  personnel  R&D  end 
products  would  also  be  useful  as  a starting  point  for  implementation  of  JPA 
technology. 

COL  Grossel’s  points  on  product  improvement  measures  for  new  and 
existing  system  development,  operational  readiness,  and  availability  to  help 
war-time  surge  capability,  and  the  need  to  increase  standardization  wherever 
possible,  reflects  the  larger  picture  that  JPA  technology  must  address  as 
well  to  ensure  eventual  implementation. 

Implementation  Research 

In  conclusion,  there  was  little  doubt  in  the  minds  of  the  conference 
participants  that  the  status  of  JPA  technology  is  at  the  point  where  major 
benefits  can  be  achieved  if  the  technology  is  implemented.  It  is  in  the 
implementation  area,  however,  where  the  R&D  technology  is  weakest  not  only 
for  job  aids  but  also  for  all  other  human  resources  products.  A conference 
that  started  with  the  question,  "Is  JPA  technology  sufficiently  developed 
for  implementation?",  ended  with  the  conclusion  that  research  is  most  needed 
on  Implementation  itself.  This  was  most  succinctly  summarized  by  Dr.  Shriver 

We,  the  human  resources  research  community,  are  in  an 
implementation  area  we  are  not  prepared  to  deal  with.  V/e 
have  a technology  base  but  not  enough  people  who  have  the 
technology  to  get  implementation  accomplished  correctly. 

The  problem  is  research  fn  implementation  itself. 
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The  selected  discussion  comments  below  followed  Dr.  Collin's  paper  "Some 
Persepctives  on  the  Job  Performance  Aids  Technology  Base." 

Mr.  Klesch:  In  the  1930s,  work  centered  on  the  job  simplification  con- 

cept. Then  there  was  work  on  actual  tasks  in  the  40s  and  later,  in  the  50s, 
we  started  moving  on  to  research  areas  for  such  things  as  motivation,  work 
fulfillment,  self-enrichment,  and  job  satisfaction.  But  it  seems  to  me  that 
with  the  Air  Force  application  of  what  we  call  JPAs  today  we  are  still  back 
with  the  job  simplification  concept. 

Dr.  Shriver:  Some  experimenters  have  done  task  simplification  when  they 

made  JPAs,  others  haven't,  but  there  is  a payoff.  You  can  get  an  effect 
whether  you  make  your  task  analysis  to  simplify  the  job  or  just  to  describe 
it  more  accurately.  There  are  several  effects  that  different  experimenters 
have  gotten.  They  have  used  different  analytic  techniques  but  any  of  the 
techniques  work.  I think  there  are  four  or  five.  If  you  put  them  all  together 
you  get  a bigger  effect.  Sometimes  the  simplification  has  been  there,  sometimes 
it  hasn't. 

Mr,  Klesch:  There  is  a question  here  regarding  the  net  impact  on  the 

individual.  That  is,  is  he  happier,  more  satisfied  being  able  to  accomplish 
a task  that  he  may  not  have  been  able  to  accomplish  before  JPA;  or,  has  he 
been  reduced  to  a lower  status  by  following  a book  technique  so  that  in  fact 
there  isn't  job  satisfaction? 

Dr.  Shriver:  My  experience  has  been  that  most  people  are  interested  in 

doing  well.  Doing  a job  well  is  the  only  intrinsic  kind  of  motivation  there 
is  in  the  military  service.  Most  everything  else  is  expensive — promotion, 
money,  etc. — but  doing  well  at  a job  is  what  most  people  really  need. 

Dr.  Booher:  What  about  peer  recognition? 

Dr.  Shriver:  I think  that  that  is  a function  of  how  the  technicians 

happen  to  grow  up.  I went  to  school  for  a year  when  I was  in  the  Navy.  That's 
how  long  the  training  was  for  electronic  technicians  during  the  war.  If  that's 
the  way  you  did  it,  that's  what  the  senior  people  did;  and  unless  everybody 
else  has  done  that  then  you  don't  really  know.  It  seems  to  me  there  is  a 
present  opportunity  to  break  that  very  neatly  with  the  supervised  on-the-job 
instruction.  It  isn't  just  reducing  the  amount  of  time  in  school  which  is 
degrading.  The  portable  JPA  has  made  it  possible  to  deliver  the  instructional 
content  on  the  job  without  the  supervisor  turning  himself  into  an  instructor. 

Mr.  Post:  You  are  taking  an  instantaneous  snap  shot  of  the  situation. 

The  person  without  the  job  guide  who  goes  out  and  is  unable  to  perform  is  now 
able  to  perform  with  the  job  guide.  That's  great.  Now  we  have  a productive 
member  of  the  group  and  everything  is  fine.  Jack  Collins'  comment  in  his  paper 
about  Maslow's  Hierarchy  needs,  that's  only  fine  for  a certain  period  of  time. 
Then  he  lias  to  go  to  the  next  plateau.  I don't  think  lob  aids  and  conventional 
JPA  style  provide  or  accommodate  for  that  next  plateau.  The  work  that  Hal  and 
I did  a couple  of  years  ago  tried  to  combine  some  different  aid  forms  so  that 
we  had  a series  of  plateaus.  Mastery  of  the  first  plateau  automatically  pro- 
vided entry  into  the  second  level  plateau.  We  have  to  support  the  gent's 
career  progression. 
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Dr.  Collins:  There  is  a human  goals  program  which  has  a lot  of  interven- 

tion strategies  and  it  has  broad-base  approaches,  not  only  to  the  quality  of 
life  but  also  performance.  I was  also  thinking  in  terms  of  whatever  progress 
we  have  made  in  the  job  performance  area  of  trying  to  become  more  definitive 
about  what  accounts  for  so  much  of  the  variance  that  we  haven't  been  able  to 
deal  with  historically.  In  trying  to  look  at  some  dimensions  in  job  perform- 
ance in  some  broader  context  than  a skill  level  kind,  I think  there  is  know- 
ledge available  which  (if  only  from  the  job  design  literature)  deals  with 
some  of  these  other  dimensions. 

Mr.  Post:  In  question  No.  3 you  talked  about  the  inclusion  of  group 

requirements . 

Dr.  Collins:  One  of  the  things  that  prompted  me  to  suggest  that,  in  addi- 

tion to  what  appears  in  the  literature  generally,  is  the  recent  action  of  the 
defense  science  board.  The  board  published  a rather  strong  report  on  the  need 
for  improvements  in  training,  including  especially  team  training.  I think  there 
is  a lot  of  interest  at  the  moment  in  this  area  of  team  training  and  implicit 
is  team  performance.  What  have  we  done  in  the  context  of  the  historical 
developments  of  JPAs  to  support  teams  as  distinct  from  individuals? 

Mr.  Post:  The  hands-on  aspects  are  obviously  personal  and  individual  but 

I think  that  it  has  to  be  a group  effort.  You've  got  to  get  the  most  from 
your  entire  work  force  whether  they  are  inexperienced  or  experienced.  In  that 
sense  I think  they're  a team  but  you  are  not  performing  in  a coordinated  two- 
man  type  fashion. 

Dr.  Shriver:  Technically  it  is  not  a team  sport.  Not  that  whenever  there 

is  a group  of  people  you  don't  attend  to  those  aspects  with  jPAs. 

Dr.  Collins:  Are  you  saying  maintenance  is  not  a team  function  at  the 

behavioral  level? 

Dr.  Booher:  What  about  the  type  of  situation  where  a man  is  down  in  a 

submarine  somewhere  and  is  trying  to  repair  something  in  a tight,  dark  environ- 
ment? How  about  the  use  of  audio  communication  where  someone  else  reads  the 
manual  or  guide  while  the  other  man  does  the  work? 

Dr.  Shriver:  I think  that  that  is  a trivial  case  where  one  person  is  a 

repeater.  It  could  have  been  done  with  an  audio  cassette.  It  is  not  inter- 
action; it's  just  a channel  of  communication.  I mean  team  in  terms  of  where 
you  have  to  do  this  while  I push  up  on  that,  the  tradition  of  group  dynamics. 
Four  people  on  the  corner  of  a box  and  each  one  must  keep  it  level  so  that  the 
ball  in  the  center  doesn't  roll  out  of  its  position.  Joint  interaction  among 
them  all.  I don't  see  that  in  maintenance  JPAs  definitely  are  changing  our 
old  notions  on  how  things  should  be  compartmentalized.  My  feeling  is  that 
JPA  makes  everybody  equal  to  go  out  and  do  the  job  with  no  training.  Follow 
the  instructions  and  do  the  job.  If  its  a calibration  job,  you  just  follow  the 
instructions.  Therefore,  everybody's  job  duties  expand  greatly. 

Dr.  Booher:  I am  very  concerned  that  we  may  not  have  an  adequate  data 

base  to  do  a demonstration  on  much  other  than  tech  data  kinds  of  things.  It 
seems  to  me  that  JPAs  have  always  been  defined  (the  way  we  use  them)  as  infor- 
mation aids.  Are  we  thinking  of  a JPA  in  terms  of  anything  other  than 


information  aids?  The  other  main  thing  I have  heard  addressed  today  is  job 
design.  There  seems  to  me  to  be  other  categories  like  tools  which  aid  physical 
areas.  Another  category  which  may  be  considered  a job  aid  is  the  automatic 
test  equipment. 

Dr.  Blanchard:  There  may  be  many  other  kinds  of  aids.  There  could  be 

psychomotor  aids  also.  There  could  be  visual  information  aids  in  the  case  of 
a situation  where  you  facilitate  an  alignment,  calibration,  etc. 

Mr.  Johnson:  When  you  finally  decide  that  you  are  going  to  need  a partic- 

ular type  of  JPA,  many  other  decisions  have  already  been  made  which  restrict 
the  choice  of  the  JPA  maker.  Specifically,  take  a new  system,  critical  design 
reviews  have  already  been  held,  and  at  this  point  in  time  the  decision  has 
been  made  to  the  degree  of  built-in  test.  The  decision  has  been  made  that 
certain  kinds  of  test  equipment  are  going  to  be  used.  A decision  has  been 
made  as  to  the  kind  of  tasks  that  have  to  be  performed  in  terms  of  the  hardware 
that  you've  designed.  Very  little  is  left  for  the  JPA  technologist  to  con- 
tribute and  say  we  are  going  to  do  this  or  not  do  this.  I would  like  to 
suggest  that  we  may  want  to  concentrate  on  what  JPA  technology  is  going  to 
assist . 


Dr,  Blanchard:  The  reason  we  are  here  is  precisely  because  of  that  prob- 

lem. We  don't  have  a solid  technology  that  has  been  organized  in  the  frame- 
work that  the  design  process  can  use.  If  we  had  something  like  that  then  we 
have  a chance  of  affecting  this  process.  We  have  to  get  this  technology  in 
the  R&D  institutions.  When  people  design  weapons  systems,  they'll  have  a means 
for  considering  tradeoffs  within  personnel  training,  maintenance,  and  the  use 
of  JPA  technology.  We  don't  have  that  tool  now.  We  can't  hold  it  out  or  we 
can't  introduce  it.  We  can't  interject  ourselves  into  that  process  because 
we  don't  have  anything  to  sell.  That  is  recognized. 

Dr.  Collins:  Stay  within  the  realm  of  maintenance  and  try  to  define 

within  at  least  those  parameters  what  kinds  of  technology  are  available. 

Dr,  Booher:  I think  the  one  big  important  area  is  job  design.  To  really 

make  the  job  aid  work  we  have  to  consider  other  characteristics  of  designing 
the  job  for  the  individual.  We  really  have  not  built  the  whole  literature  of 
what's  going  on  in  job  design  into  our  technology  base. 

Mr.  Post:  Let  me  digress  for  a moment  to  Rowan’s  report.  Somebody  else 

seems  to  think  that  it  was  relatively  negative,  but  I think  it  was  relatively 
honest.  As  I recall,  I think  he  reviewed  some  14  or  15  field  experiments  that 
evaluated  information  aids  of  one  type  or  another.  I think  more  than  half 
were  positively  assessed  by  Rowan,  maybe  as  many  as  two-thirds.  He  made  com- 
ments about  the  experimental  design  and  the  rationale  of  the  test  with  regard 
to  some  others.  I think  of  it  as  a positive  appraisal. 

Dr.  Blanchard:  I think  we  are  finding  it  difficult  to  carry  out  experi- 

mental controls  in  the  traditional  sense.  Experimental  design  in  the  applied 
situations  is  a difficult  process.  There  are  many  variables  that  have  to  be 
controlled. 

Mr.  Post;  In  terms  of  some  of  the  JPA  technology,  I think  we  are  pretty 
well  along  and  ready  to  apply.  I think  some  particular  sectors  of  that  tech- 
nology might  have  some  holes,  but  it  would  be  useful  to  get  it  out  into  the 


field.  I think  the  answer  is  both  yes  and  no.  Yes,  for  some  purposes  and  no 
for  others;  and,  in  which  case,  we  have  to  go  into  some  R&D  phases.  I think 
that's  one  of  the  things  that  the  Navy  has  not  till  recent  months  done  very 
well.  We  have  had  no  underlying  technology  base  development.  In  the  50s,  what 
was  being  done  about  JPAs  that  we  can  now  draw  upon?  Hardly  anything.  In  the 
60s,  what  was  done?  Most  of  the  inputs  for  this  whole  program  came  with  the 
Air  Force  PIMO  Program.  I think  we  need  both.  We  have  to  get  some  portions 
of  technology  out  into  the  field  and  draw  the  benefits  from  it.  We  have  to 
continue  to  develop  the  technology  at  the  same  time. 

Mr.  Johnson:  The  technical  manual  publications  system  as  it's  entrenched 

today  is  not  really  ready  to  accept  JPAs. 

Dr.  Collins:  Is  there  any  special  kind  of  test  and  evaluation  technology 

unique  or  applicable  to  JPA  evaluation? 

COL  Grossel:  The  change  is  to  really  try  and  get  the  program  managers 

to  take  more  of  the  life  cycle  cost  approach  in  developing  systems.  They  are 
not  going  to  be  on  watch  when  the  system  is  operational,  but  there  Is  definitely 
a very  strong  drive  to  force  the  program  manager  to  include  life  cycle  costs 
in  the  design  of  the  system.  There  is  a new  instruction  that  came  out  (5000.2) 
on  acquisition  of  systems.  There  is  a requirement  in  there  at  the  DSARC  1,  2, 
and  3 level  to  have  a one-page  logistics  annex. 

Mr.  Johnson:  The  JPA  tool  includes  something  that  is  already  going  on 

that  we  should  make  ourselves  aware  of — the  maintenance  engineering  analysis. 
Then,  getting  to  your  second  point,  how  do  we  use  the  tool?  I suggest  that 
the  way  to  make  this  thing  work  is  to  get  the  results  of  our  analyses  included 
and  made  a part  of  the  engineering  production  drawings.  For  example,  right 
now  every  engineering  production  drawing  that  goes  off  to  that  shop  includes 
one  line  for  a schematic  diagram.  It  strikes  me  that  we  could  have  another 
line  that  says  JPA  technology  procedure  number  such  and  such  for  each  separate 
procedure  that  was  developed  for  that  black  box.  You  could  make  it  follow  the 
design  process  that  we  enter  and  it  becomes  part  of  the  engineering  production 
drawing.  Once  that  is  done,  then  you've  got  a self-healing  kind  of  thing 
because  the  inspector  who  is  testing  the  equipment  will  insist  that  that  line 
of  that  drawing  be  performed. 

Dr.  Blanchard  I think  that  we've  got  to  get  in  even  earlier  than  that. 

We  talked  about  trade  ff  analysis  early  in  system  design  conception  where 
you  are  looking  at  various  alternative  ways  of  performing  certain  functions. 

I think  we  have  to  be  in  there  because  that's  where  you  are  looking  at  per- 
sonnel variables  and  training  variables.  Once  you  get  into  the  maintenance 
engineering  analysis  you  have  a fairly  fixed  system  concept.  That  may  he  a 
bit  late.  I'd  like  to  shoot  for  something  a little  earlier. 

Dr.  Shriver:  The  pattern  I see  over  the  years  is  to  stop  talking  about 

such  things  separately,  such  as  this  is  training,  this  is  documentation,  and 
this  is  training  aids.  It  is  all  one  thing.  Allocating  to  those  old  cate- 
gories sounds  funny  to  me  at  this  point.  It  seems  to  me  all  categories  are 
being  merged  into  one  thing.  The  old  assumption  was  that  training  is  supposed 
to  provide  the  individual  with  certain  knowledges.  The  book  is  supposed  to 
describe  the  equipment.  Then  you  bring  the  two  together  and  the  individual 
figures  out  what  to  do.  A new  approach  is  to  analyze  the  whole  thing  and  figure 
out  what  he  is  supposed  to  do  and  tell  him. 


Mr.  Joyce:  When  there  is  a substantial  string  of  things  that  a guy  has 

to  do,  it  doesn't  lend  itself  very  well  to  his  following  an  aid  in  the  time- 
limited  situation.  While  the  steps  that  he  performs  may  be  correct,  there 
are  many  situations  where  he  is  going  to  have  to  learn  to  do  that  thing 
prior  to  the  time  he  comes  to  it  in  the  job  aid. 

Dr.  Shriver:  All  tasks  can  be  proceduralized.  I don't  know  if  it  is 

the  most  efficient. 

Dr.  Blanchard:  It  seems  we  are  not  looking  down  the  road  far  enough  in 

terms  of  what  happens  to  this  guy.  We  may  have  some  problems  down  there. 

We  have  looked  at  certain  variables  and  our  optimization  models  say  if  we  can 
do  these  kinds  of  things,  we  can  achieve  this  kind  of  payoff  and  it  looks 
pretty  good.  However,  if  we  stretch  this  down  the  road  three  years  and  look 
at  channels  of  advancement  and  what  the  personnel  structure  might  look  like, 
it  begins  to  get  shaky.  There  are  other  kinds  of  problems  that  have  to  be 
considered  in  terms  of  how  we  use  JPA  technology. 

Dr.  Shriver:  You  can  put  concepts  in  a book  for  him  to  memorize  and 

carry  around  in  his  head,  that  is,  block  diagrams  and  things  like  that 
which  are  not  fully  proceduralized.  Instead  of  an  instructor  saying  it, 
the  book  becomes  the  communication  device. 

Dr.  Shriver:  You  can  deal  with  paper  and  that  is  real  world.  Technical 

documentation  the  world  knows  about.  The  effects  that  have  been  documented 
are  there  because  of  types  of  analysis,  not  whether  it's  displayed  on  a CRT, 
or  a piece  of  film,  or  on  a piece  of  paper.  CAI  gets  into  tough  program- 
ming. Also,  paper  is  the  cheapest  way  to  do  it. 

Dr,  Collins:  I would  just  like  to  make  one  comment  in  terms  of  the 

task  analysis  side  of  the  development  of  JPAs.  I would  suggest  that  we  seri- 
ously consider  as  an  option  a behavioral  requirements  approach  as  distinct 
from  behavioral  descriptive  approach.  In  the  behavioral  requirements  approach, 
of  course,  the  emphasis  is  on  acceptable  levels  of  standards  of  performance  and 
it  consists  of  both  a skill  and  a training  dimension.  It  also  seems  clear 
under  the  broader  question  that  maintenance  is  probably  the  demonstration 
area  that  we  would  want  to  pursue. 

Dr,  Shriver:  Just  to  pick  up  what  Jack  said  on  the  last  point,  maintenance 

certainly  seems  to  be  just  the  area  to  stick  to;  don't  broaden  it.  There  is 
plenty  of  payoff  there.  We  know  what  we're  doing.  Stick  to  what  you  know.  I 
think  the  technology  base  is  strong  enough  there.  People  all  over  the  country 
are  doing  these  things  now.  The  analysis  results  can  be  put  in  a book  easily 
enough  but  when  you  get  to  the  technology  of  putting  it  in  some  other  form 
than  in  a book,  e.g.,  CAI,  etc.,  I think  it  is  getting  risky. 

COL  Grossel:  I am  still  not  adequately  satisfied  in  my  own  mind  that  we 

have  tackled  the  definition  of  JPA,  although  a lot  of  things  have  been  discussed. 
I think  that  there  are  really  two  main  areas  where  we  can  apply  the  technology 
today.  I think  I agree  with  what  Ed  said  that  book  technology  seems  to  be  sort 
of  defined  and  there  seem  to  be  some  concepts  which  people  can  be  sold  on  the 
idea  of  applying  even  without  further  data.  Further  data,  of  course,  will  help 
the  speed  and  depth  of  implementation.  I think  the  biggest  thing  right  now  is 
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what  you  might  call  product  improvement — implementation  on  existing  systems. 

If  you  go  product  improvement,  you  don't  have  all  these  tradeoffs  of  ATE  vs. 
job  aids  vs.  other  support  test  equipment.  I think  you  definitely  take  a 
different  approach  although  your  technology  base  may  have  a lot  of  common 
elements  in  it.  If  you  go  design  of  new  equipment,  you  bring  up  all  the 
things  that  Frank  was  mentioning  in  the  LSA  analysis  and  the  MEA  analysis. 

An  awful  lot  of  things  have  to  be  considered,  and  one  of  the  big  things  in 
that  area  that  I see  is  a need  to  get  a better  handle  on  the  tradeoff  decisions. 
I think  too  many  trades  in  human  resources  are  made  in  heads  of  one  or  a group 
of  engineers,  and  they  are  undocumented.  They  use  their  experience  and  say  okay 
this  is  the  way  we  are  going  to  go.  We  are  going  to  use  ATE  to  do  such  and 
such  without  bringing  those  decision  points  to  the  surface.  What  grounds  were 
considered,  what  were  the  objectives,  what  were  the  constraints,  and  then  what 
were  the  decisions  made?  Without  all  that  being  documented  adequately,  I 
don't  think  we  can  ever  make  an  improvement.  We  have  got  to  make  them 
visible.  We  have  to  have  a path  that  we  can  trace  back  and  also  get  his- 
torical purposes  of  this  system.  What  approach  did  they  use,  and  were  they 
successful  or  unsuccessful?  With  respect  to  both  new  equipment  and  product 
improvement,  I think  we  have  got  to  define  what  we  see  as  the  purpose  of  the 
job  aid.  Is  it  just  for  operators  and  maintainers?  In  addition,  is  it  to 
provide  some  sort  of  training,  probably  on  the  job?  Don't  eliminate  training, 
but  put  a new  man  out  there  who  can  train  on  the  job  and  develop  some  of  this 
knowledge  in  the  head.  We  can  take  lesser  initial  training;  but,  in  addition 
to  the  operating  and  maintaining,  we  have  got  upgraded  skills  to  consider. 
Getting  into  the  hierarchy  of  needs,  possibly  through  this  training,  you  can 
upgrade  his  skill  levels,  upgrade  his  self-image,  upgrade  his  actual  contri- 
bution to  the  mission.  If  that's  the  purpose,  well  maybe  some  of  the  simpli- 
fied proceduralized  approaches  aren't  the  best  or  at  least  the  only  answer. 

There  would  be  other  things  that  can  help  him  upgrade  his  skills.  I think 
that  anything  you  can  give  to  someone  and  have  it  essentially  done  by  "a 
monkey"  doesn't  really  do  much  to  upgrade  the  skills  of  that  person.  I think 
we  have  also  got  to  consider  readiness  and  availability  and  operational  needs. 

We  have  got  to  look  at  training  vs.  wartime  in  the  face  of  surge.  Right  now, 
let's  face  it,  we  are  in  a training  situation.  We  are  training  for  defense 
or  whatever  you  want  to  call  it.  Shouldn't  we  really  convince  these  people 
to  try  out  some  new  concepts  right  now  when  we're  really  not  at  war?  We  are 
certainly  not  going  to  get  any  of  these  things  when  we  are  at  war,  because 
then  we're  there  to  shoot  guns  or  drop  bombs  or  ward  off  the  enemy  and  so  on. 
Maybe  some  of  these  new  approaches  can  also  help  us  in  the  surge.  If  they  can, 
in  fact,  in  a real  environment  take  a new  person  with  minimal  training  and 
bring  him  up  to  skill  quickly  and  provide  for  on-the-job  training,  provide 
some  support  to  the  mission,  provide  for  upgrading  of  skills  and  still  retain 
attention  to  the  personnel  aspects  of  it,  we  increase  the  surge  capability  of 
our  forces. 

Mr.  Post:  I think  in  the  interest  of  a solid  demonstration  for  the  program 

we  ought  to  define  JPA  tools  as  the  things  we  know  most  about  which  comes  in 
two  forms.  One  is  the  proceduralized  form  which  tells  the  user  what  to  do  and 
how  to  do  it.  The  other  form  I've  been  calling  "deductive,"  and  it  leaves  the 
user  to  go  on  his  own  inferences  as  to  what  to  do.  I think  that  we  should 
restrict  ourselves  to  these  two  basic  aid  forms,  and  the  thing  we  need  to  do 
in  the  program  is  to  devise  a methodology  that  tells  us  when  and  how  to  use 
either  or  both  or  combinations  of  these  two  basic  forms.  I think  we  should  do 


what  we  know  most  about  and  the  thing  that  our  "audience,"  whoever  that  is,  is 
predisposed  to  accept.  I think  that  is  maintenance.  Rather  than  branch  out 
into  a new  application,  let's  stick  with  the  firm  definition  or  demonstration 
of  what  we  know  in  a field  that  most  people  know  about. 

Mr.  Joyce:  I agree  Just  about  entirely  with  what  Ted  said.  The  areas 

in  which  most  of  us  in  the  room  are  most  familiar  are  construction  and  use  of 
those  two  kinds  of  aids.  I think  we  really  do  have  a technology  for  building 
both.  I think  we  can  look  backwards  from  those  concepts  to  how  we  could 
interact  with  people  who  are  in  a position  to  design  the  jobs  and  come  up  with 
more  optimal  job  designs  that  include  the  use  of  aids.  I think  that  the  tech- 
niques that  we  have  right  now  for  building  either  of  the  kinds  of  aids  we  have 
talked  about  can  deal  with  any  kind  of  mix  of  job  designs.  We  tend  to  let 
ourselves  be  driven  by  the  way  the  job  presently  is  designed,  and  we  build  the 
aids  to  reflect  the  job  as  it  is  which  may  not  be  optimum.  We  get  in  trouble 
that  way.  We  wind  up  with  our  own  interface  problem  because  we  have  been 
driven  by  an  existing  job  design.  I agree  that  the  demonstration  that  comes 
out  of  this  project  is  going  to  have  to  consider  a mix  of  these  kinds  of  aids 
for  no  other  reason  than  to  deal  with  the  problem  of  building  career  struc- 
tures for  people  that  they  are  willing  to  tolerate.  We  have  by  virtue  of 
the  structure  of  some  of  our  experiments  led  the  world  to  believe  that  we've 
got  some  kind  of  peculiar  ideas  about  what  our  aids  are  supposed  to  do.  We 
have  left  in  our  wake  some  bad  feelings  because  we  failed  to  try  to  explain 
to  people  after  we  left  what  they  are  supposed  to  do. 

Mr.  Johnson:  First  I'd  like  to  say  that  we  should  stick  in  the  field  of 

maintenance  because  I think  we  know  most  about  that.  I'd  like  to  say  that  we 
should  go  ahead  and  use  the  existing  technology  which  is  the  proceduralized 
aids.  If  we  are  going  to  really  be  effective  in  a demonstration,  we  have  got 
to  look  at  the  maintenance  system  as  such  and  identify  what  kind  of  aid  we 
are  talking  about.  Obviously,  we  cannot  say  that  a JPA  is  going  to  do  every- 
thing that  the  existing  technical  data  does  now,  so  we  should  find  out  how 
the  JPA  complements  and  uses  information  that  is  already  available  in  the 
technical  manual  system.  If  we  don't  do  this  systems  maintenance  aid  design, 
we  are  still  looking  at  a little  tiny  tree  in  the  big  woods.  I do  believe 
that  if  we  are  going  to  really  demonstrate  that  we  have  a technology  base,  we 
have  got  to  first  Identify  through  some  kind  of  maintenance  engineering 
analysis  where  the  JPA  fits  in  the  maintenance  system;  and  than  go  ahead  and 
produce  the  JPAs  for  those  areas  that  are  most  applicable  and  profitable. 

Mr.  Klesch:  I think  your  demonstration  should  attempt  to  convince  others 

and  ourselves  of  the  ability  of  the  JPAs  to  indeed  make  an  impact  on  life- 
cycle  costs.  In  that  regard  I think  the  technology  base  for  development  exists. 
Technology  for  proper  assessment  of  other  impacts  does  not  exist  and  has  not 
been  utilized.  I think  one  of  the  other  areas  that  we  should  also  be  inter- 
ested in  is  the  payoffs  in  the  long-term  on  personnel,  for  example,  the  psycho- 
logical effects.  So  far  we  have  only  experimentally  demonstrated  how  we  want 
the  man  to  use  the  JPA  and  how  we  want  him  to  learn.  The  question  is,  how,  in 
fact,  will  this  occur  in  the  real  world?  We  may  be  off  base.  I think  you'll 
have  wasted  a lot  of  good  money  to  take  one  type  for  a month  or  2-month  demon- 
stration. I don't  think  that's  going  to  yield  any  payoff  that  will  help  us.  We 
need  to  have  something  more  in-depth.  Believe  it  or  ncrt,  when  we  are  talking 
about  the  impact  on  personnel,  we  are  talking  about  1 to  5 years. 
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SESSION  2 — Wednesday  afternoon,  February  23,  1977 

The  selected  discussion  comments  below  followed  the  paper  of  Mr.  Post  on 
"Methodology  for  Selecting  Optimum  JPA  Techniques." 

Dr.  Booher:  One  question  about  your  model,  Ted.  Are  we  to  assume  that 

this  is  the  kind  of  thing  that  the  program  manager  would  have  available? 

Some  of  these  items  we  would  have  available,  but  I wonder  about  the  things 
like  span  of  supervision.  At  the  decision  stage  he  may  assume  he's  going  to 
have  a narrow  span  of  supervision  but  in  real  life  it  turns  out  differently. 
Another  comment  is,  perhaps  one  of  the  objectives  of  the  program  manager 
would  be  to  increase  the  skill  level  or  utilization  of  his  people,  rather  than 
having  it  an  established  fact  that  he  always  has  low-skilled  people.  He  may 
feel  that  an  aid  would  allow  him  to  use  lower-skilled  people. 

Mr.  Post:  Diagnostic  technique  would  be  a difficult  thing  for  him  to 

predict  at  the  early  stages  of  system  design.  I guess  we've  got  to  give  the 
program  manager  some  sort  of  generalized  model  that  will  allow  him  to  run 
through  these  kinds  of  considerations.  Whether  it's  a set  of  guidelines  that 
we  try  to  develop  to  reduce  these  considerations,  I don't  know  at  the  moment. 
I've  talked  to  a couple  of  program  managers,  and  I guess  the  feeling  is  that 
they  are  not  going  to  go  out  on  a limb  for  some  fly-by-night  idea.  You've  got 
to  give  them  some  concrete  data  and  relatively  good  assurances.  There  are  too 
many  other  safer  options  open  to  them.  I don't  have  any  answer  to  that  par- 
ticular question,  but  this  is  an  important  issue. 

Dr.  Collins:  Ted,  about  that  slide  on  conditions  versus  impacts,  do  you 

have  some  data  with  regard  to  the  assumption  of  the  relationship  between  those 
conditions  and  impact  areas?  Did  you  generate  some  data  at  some  points? 

Mr.  Post:  In  some  cases,  yes,  I do  have  some  data,  but  in  a lot  of  cases 

I do  not.  It’s  hopefully  a rationale  that  was  presented  to  support  the  asso- 
ciation. The  business  about  subordination,  for  example,  is  really  based  on 
the  amount  of  information  that  the  aid  would  have  to  contain — the  more  infor- 
mation, the  more  complicated.  This  is  supported  to  some  extent  in  the  litera- 
ture. In  the  example  about  cleanliness  dealing  with  the  medium  you  use  to 
present  information,  it  is  just  something  simple  like  a laminated  page  or  a 
cleanable  viewer.  There  is  not  much  there  you  would  have  to  go  to  research 
literature  to  support.  Take  work  space,  for  example.  There  has  been  some  work 
done  on  cramped  work  space  and  suitability  of  head-mounted  devices.  Maybe  for 
about  one-half  to  two-thirds  of  the  conditions,  I could  point  to  the  literature 
and  say  that's  where  I got  my  idea.  The  other  third  I might  have  to  say  I made 
it  up . 


Dr.  Collins:  Is  there  any  differential  weighting  here  with  regard  to  how 

these  are  used  or  combined?  It's  not  going  to  be  made  on  a single  condition  by 
the  program  manager.  Do  you  have  some  data  that's  going  to  relate  to  how  those 
will  be  weighted  and  combined?  Do  you  plan  to  get  any? 

Mr.  Post:  The  plan  to  get  it  is  probably  the  better  description.  For 

example,  I don't  know  what  the  relative  contribution  would  be  between  hardware 
complexity  factors  and  supervision  purposes.  I'd  like  to  go  out  and  establish 
that  experimentally. 


Dr.  Blanchard;  We  are  looking  for  some  way  of  quantifying  the  performance 
levels  that  are  delivered  under  various  combinations  of  factors.  I have  those 
divided  up  into  environment,  activities,  and  personnel  characteristics.  How 
sensitive  is  the  task  of  media  selection?  Are  we  going  to  come  up  with  a 
directive  or  deductive  approach  to  specify  what  the  aid  would  look  like?  The 
question  is  just  how  sensitive  is  that  process  to  variations  in  those  factors. 
Also,  I don't  like  the  use  of  labels,  FOMM,  SIMM,  etc.  In  many  cases,  it 
covers  up  important  properties  for  which  we  want  to  account.  Ultimately  we 
should  tie  into  your  conditions  matrix  and  be  able  to  model  payoff.  We  can 
start  with  ordinal  property  data  and  hope  some  day  to  improve  it.  Starting 
with  ordinal  data,  we  can  give  the  decision  maker  some  idea  of  what  his  pay- 
offs might  be.  The  weights  might  be  gross,  but  at  least  they  can  give  him 
direction. 


Mr.  Post:  Let's  tailor  the  aid  to  the  depot  situation.  Let's  tailor  the 

aid  to  the  work  center.  Or  let's  tailor  the  aid  to  the  bows  of  the  ship  or 
the  automatic  boiler  control  system.  Let's  not  use  one  technique  and  hope 
that  it  works  equally  well  in  all  those  circumstances  because  those  circum- 
stances differ. 

Dr.  Collins:  If  you  look  at  an  analogy  in  human  factors  engineering 

where  you  are  trying  to  make  an  impact  at  the  conceptual  stages,  you  are 
dealing  principally  with  data  and  principles  of  human  engineering,  and  habit- 
ability applied  against  some  classes  of  problems  that  are  perceived  to  exist 
within  this  new  system.  The  level  of  generality  of  the  class  of  problem  becomes 
important  in  trying  to  relate  it  to  some  principles  and  guidelines  for  making 
decisions  about  whether  it's  going  to  be  a man  in  the  system  or  piece  of 
hardware  in  the  system.  If  one  were  trying  to  think  of  the  DSARC  process, 
you  would  have  to  think  in  terms  of  a handbook  which  has  some  principles  for 
selection.  One  of  the  things  this  points  out  is  the  decision  making  process 
in  the  whole  DSARC  process  relates  to  where  job  performance  aids  ought  to  be 
considered,  what  kinds  of  information  are  important,  etc.  That  is  an  area 
where  nothing  has  been  done  to  identify  what  those  decisions  are  and  where  in 
the  process  they  should  occur. 


Dr.  Blanchard:  There  is  an  interesting  point  here  concerning  the  perennial 

struggle  of  the  human  factors  engineer  for  acceptance  and  consideration  in  the 
design  process.  I think  we  are  proposing  to  do  one  thing  that  the  human  factors 
community  has  failed  to  do;  that  is,  to  bring  the  decision  maker  some  kind  of 
established  set  of  payoff  expectancies.  To  get  attention,  we  have  to  be  per- 
suasive. In  order  to  be  persuasive,  we  have  to  bring  the  decision  maker  some 
kind  of  formalized  model  that  has  a data  base  so  he  can  see  what  his  payoff 
expectancies  are. 

Dr,  Collins:  That's  one  of  the  objectives  that  the  human  factors  engi- 

neering integration  project  is  to  produce.  Produce  the  handbook  which  will 
deliver  to  the  engineer  information  to  tell  him  what  the  payoff  is  if  he  goes 
with  these  techniques  or  these  kinds  of  decisions  in  human  factors  engineer- 
ing. 

Dr.  Booher:  There  is  another  question  I want  to  raise  under  this  topic. 

It  is  related  to  cost  variables.  We  feel  we  don't  have  a lot  of  cost  data  or 
a way  of  getting  at  cost  data  to  make  this  tradeoff  even  after  we've  gone 


through  the  algorithm  for  the  other  reasons,  showing  its  payoff  for  operational 
readiness,  personnel  turnover,  etc. 

Dr.  Shrlver:  All  I recall  seeing  on  cost  was  the  cost  to  produce  the 

manuals  not  the  payoff  in  personnel  time.  They  are  probably  different  by  a 
magnitude  of  ten.  If  you  get  even  a tenth  of  the  savings  on  the  personnel 
effectiveness  side,  it's  worth  doing.  The  analysis  time  is  the  expense.  What- 
ever format  you  put  it  up  in  is  relatively  unimportant  in  cost. 

Mr.  Post:  The  only  indicator  I have  seen  is  the  Air  Force  work,  and  it 

seems  that  they  are  indicating  an  order  of  magnitude  cost  difference  between 
conventional  materials  and  fully  proceduralized  JPAs. 

Mr.  Johnson:  We  produced  MDCs  for  the  F4J  and  they  cost  the  government 

a little  over  $1000  a page.  Don't  forget  they  were  generally  two-and-a-half 
folds,  11  x 17 — tremendously  compact  information.  To  support  your  $2900 
figure,  which  is  the  top  figure  you  have,  Ted,  these  MDCs  we  produced  were  pro- 
duced from  existing  technical  manuals  and  technical  manual  data  (which  is  not 
the  point  you  made) . Conceivably  by  the  time  you  went  through  the  first  cut, 
the  various  iterations,  the  manuscript,  validation,  and  vertif ication,  you 
could  very  well  be  up  to  $3000  a page.  The  question  is,  are  you  producing 
them  from  engineering  drawings,  data,  specs,  etc.,  or  are  you  producing  them 
from  existing  technical  manuals?  Most  JPAs  have  been  produced  from  existing 
technical  manuals. 

Dr.  Shriver;  It  depends  on  how  you  look  at  data.  As  I look  at  it,  it's 
about  the  same  cost;  and  it  will  come  out  about  the  same  in  the  long  run  after 
people  get  used  to  it.  That  was  also  true  on  all  the  tank  turrets  that  they 
produced  new  ITDT  JPAs  for.  It  is  about  the  same  ball  park.  To  a large  extent, 
you  don't  go  to  engineering  information  to  create  the  JPAs,  you  go  to  behavioral 
performances . 

Dr.  Shriver:  The  training  people  have  gotten  to  the  program  managers. 

Validation  is  what  training  people  want.  They  want  to  see  that  people  can 
actually  perform  and  get  products.  The  training  command  assures  that  the 
program  manager  pays  attention  to  validation.  It  is  required  on  the  charts, 
but  they  are  the  ones  that  are  watching  to  see  that  it  gets  done.  It  seems  that 
it  is  up  to  us  to  change  industry.  Three  months  after  we  completed  the  ITDT 
specification  for  JPAs,  it  was  out  on  the  street  for  procurement  of  five  major 
systems.  There  were  eight  or  ten  companies  in  the  country  who  were  ready  to 
go.  Management  does  miracles  when  it  has  to  for  business. 

Mr.  Johnson:  Whole  empires  within  DoD  have  their  own  security  blanket 

to  worry  about.  We  in  industry  have  to  stay  alive.  We  are  going  to  go  with 
our  leaders.  Management  in  industry  will  get  it  straightened  out.  They  can 
do  it.  But  the  Government  has  got  to  ask  for  it. 

V. 

Dr.  Collins:  The  quality  assurance  points  are  not  dependent  upon  the 

hardware  availability.  That's  not  part  of  the  technology  interface  required 
to  conduct  it. 

Mr.  Post:  It  seems,  Frank,  what  you're  really  saying  is  that  there  are 

so  many  problems  regarding  quality  assurance,  validation,  verification,  etc.. 
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that,  in  fact,  we  are  transferring  a lot  of  responsibility  to  the  user  in  the 
field.  If  that  is  the  case,  then  we  ought  to  concentrate  on  firing  up  on  the 
feedback  loop. 

Dr.  Blanchard;  We  need  to  have  some  way  of  interacting  with  the  informa- 
tion in  the  tech  data  base.  That  is  going  to  influence  how  we  go  about  build- 
ing JPAs,  or  how  we  go  about  trading  off  with  training.  This  whole  process 
is  based  on  a temporal  framework  with  the  data  coming  out  of  the  engineering 
development  process. 

SESSION  3 — Thursday  morning,  February  24,  1977 

The  selected  discussion  comments  below  followed  the  papers  given  by 
Mr.  Johnson,  "Problems  in  Procuring,  Producing,  and  Evaluating  JPAs"  and 
Mr.  Joyce,  "User  Problems  in  JPA  Utilization." 

Dr.  Collins;  How  accurate  can  we  really  expect  to  be  in  JPA?  Have  we 
promised  too  much  accuracy?  That  is  a technological  issue  that  seems  to  be 
critical  to  dealing  with  the  problem,  the  fact  that  a step  has  been  left  out 
or  not  left  out.  I think  it’s  even  more  fundamental  than  that  and  that  is 
the  implementation  of  these  processes  is  going  to  vary  organizationally, 
individually,  environmentally,  and  on  a lot  of  other  dimensions.  I think 
the  technical  question  is  can  you  cover  all  of  those?  I suspect  that's  the 
real  technical  limitation.  Realistically,  you  may  only  be  able  to  deal  with 
the  training  interface  in  this  program. 

Mr.  Joyce:  One  of  the  reasons  we  were  having  a problem  is  because  the 

decision  was  made  for  that  system  to  take  the  proceduralized  troubleshooting 
all  the  way  down.  There  was  essentially  nothing  very  well  planned  in  the 
way  of  other  backup  tech  data  and  other  more  conventionally  trained  and 
oriented  people  to  do  the  system  maintenance.  We  were  having  to  validate 
procedures  that  went  so  far  down  that  we  were  being  interfered  with  by  hard- 
ware problems.  If  we  had  chosen  to  cut  the  proceduralized  troubleshooting 
at  a higher  level  and  just  stop,  we  would  have  been  much  better  off.  By  a 
higher  level,  I would  say  to  the  board  rather  than  to  the  components  on  a 
board. 


Mr.  Johnson:  It  comes  back  to  your  point  of  where  do  the  errors  come  from? 

There  are  two  kinds  of  errors.  First  the  errors  in  the  procedures  may  have 
been  developed  because  the  problem  was  very  difficult.  To  develop  accurate 
procedures  you  have  to  have  very  good  engineering  writers  to  be  able  to  do 
all  the  logic  and  be  able  to  do  all  the  combinations  and  permutations  of 
possible  troubles  and  faults.  That  is  the  one  area,  the  other  one  relates 
entirely  to  the  hardware.  Errors  creep  in  because  of  hardware  revisions. 

If  I have  a good  set  of  drawings  in  a good  operating  environment  in  terms  of 
producing  the  equipment,  the  plant  that  uses  those  drawings  is  going  to  pro- 
duce that  equipment  to  these  drawings.  If  I then  do  my  technical  manuals, 
fully  proceduralized  JPAs  included,  to  these  drawings  and  follow  whatever 
changes  are  made  to  those  drawings  with  revisions  in  my  manuals,  then  I'm 
going  to  reduce  that  kind  of  an  error.  The  only  error  I will  have  left  of 
any  consequence  is  the  error  that  my  troubleshooting  procedures  are  not  good. 
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Dr.  Boo her:  I think  that's  a good  point  about  limitations  of  people  that 

can  do  this.  We  have  this  problem  in  every  field. 

Dr.  Collins:  That's  part  of  the  technology  base  to  me.  We  have  a cap- 

ability to  design,  develop,  test,  and  evaluate  people  limitations. 

Dr.  Booher:  Getting  good  people  to  do  the  troubleshooting  is  difficult 

and  apparently  there  is  still  a lot  of  art  to  it.  You  may  know  who  the  best 
man  is,  and  he  may  be  selected  for  some  of  the  work.  Then  you  have  all  the 
rest  of  the  men  that  have  to  do  something,  and  they  are  just  not  going  to  give 
you  the  best  product  every  time. 

Dr.  Booher:  From  what  Ed  said  and  from  what  we  know,  even  if  the  JPA  we 

select  should  cost  more,  the  PM  should  select  it  over  some  traditional  approach 
because  of  the  total  system  life-cycle  savings.  But,  if  it  gets  down  to  the 
point  where  someone  has  to  make  a decision  to  pay  more  for  something  than  he 
would  have  expected,  he  is  going  to  be  very  reluctant  to  make  that  decision. 
However,  if  right  at  that  time  you  can  show  them  the  savings  in  the  other  data 
areas,  the  decision  becomes  much  easier.  You  can  almost  make  the  decision 
based  on  performance  parameters  and  system  needs  and  not  even  have  to  consider 
the  training  and  personnel  savings.  That  is  all  extra.  There  is  an  awful  lot 
of  slop  just  in  the  data  area  itself  which,  if  better  cost  data  were  available, 
would  greatly  ease  the  decision  making  of  the  program  manager. 

Mr.  Joyce:  Even  if  we  spend  a good  bundle  at  the  front  end  on  procedural 

troubleshooting  for  the  first  enlistment  guys,  we  won't  be  spending  huge  train- 
ing bucks  on  them.  You  can  recover  the  cost  of  your  whole  JPA  production 
effort  in  the  first  couple  of  years  by  avoiding  the  cost  of  training  these 
people.  Even  if  we  still  have  that  same  kind  of  training,  we  apply  it  on  a 
much  smaller  scale  to  a smaller  group  of  people  downstream,  we're  still 
avoiding  those  costs. 

Mr.  Kleach:  The  situation  in  the  Air  Force  is  that,  in  many  of  our  main- 

tenance squadrons,  the  first-term  airman  is  indeed  performing  a great  deal  of 
maintenance.  He  is  not  just  a black  box  swapper.  He  is  indeed  performing  a 
great  deal  of  maintenance.  I think  we  need  to  dispell  some  of  our  own  myths 
here  about  who  does  maintenance. 

Mr.  Post;  If  you  go  to  a workshop  or  some  work  center  in  a maintenance 
situation,  and  you  watch  the  supervisor  in  the  way  he  handles  his  people,  one 
of  the  ways  he  gets  productive  labor  from  the  inexperienced  portion  of  his 
work  force  is  to  chuck  the  job. 

Mr.  Kleach:  If  you  have  to  operate  with  only  ten  people,  suddenly  JPAs 

take  on  a tremendous  Importance.  That  is  the  only  way  they  are  going  to 
survive.  If  you  give  them  20  men  to  do  it,  they  can  handle  it  without  job  aids. 

Mr.  Post:  One  of  the  things  we  don't  do  in  trying  to  assess  the  utility  of 

job  aids  is  to  compare  them  with  other  styles  of  operation.  I think  that  the 
technicians,  even  at  the  lowest  experience  level,  are  not  really  dummies;  and 
they  have  a great  deal  of  ingenuity.  They  can  figure  out  how  to  do  certain 


things.  In  the  eyes  of  the  user,  is  the  JPA  better,  worse,  or  the  same  as 
getting  advice  from  his  peers,  sitting  together  and  working  it  out  according 
to  their  own  ingenuity,  or  admitting  to  their  supervisor  they  don't  know  and 
ask  for  advice?  There  are  several  alternative  ways  of  getting  the  job  done. 
JPAs  have  to  compete  with  these  several  alternative  ways.  So  far  it  doesn't 
seem  to  be  competing  very  well. 

Dr.  Collins:  Task  analysis  is  faced  with  three  kinds  of  options:  (1) 

you  can  go  out  and  sample  the  operational  environment  based  on  descriptive 
information  and  you  can  then  report  on  what  goes  on  in  terms  of  structure; 

(2)  you  can  form  some  theoretical  point  of  view,  look  at  these  tasks,  and  then 
superimpose  some  principles  on  them  and  structure  them  along  some  "optimum" 
position;  and  (3)  you  can  listen  to  what  the  management  people  who  operate 
the  system  say  the  position  ought  to  be.  It  seems  that  JPA  developers  have 
to  decide  where  they  are  going  to  fit  into  those  kinds  of  options. 

Mr.  Klesch:  I think  production  control  and  acceptance  are  all  tied 

together.  Based  on  my  experience,  I think  we  do  not  have  as  much  acceptance 
as  we  would  like  because  of  the  fact  that  we  haven't  produced  solutions  to 
real  problems  that  users  have.  This  is  not  a categorical  statement  though; 
it  is  qualified.  One  qualification  is,  for  example,  on  nontroubleshooting 
tasks.  I think  we  have  convinced  users  that  we  can  give  them  much  clearer 
data,  easier  to  understand.  We've  conducted  actual  usability  studies  and 
asked  people  to  compare  this  to  the  tech  manual  and  there  is  no  question 
about  it.  We  have  not  made  the  same  kinds  of  inroads  on  troubleshooting. 

I'm  not  sure  that  the  total  troubleshooting  state-of-the-art  is  in  fact  here. 

I would  suggest  that  any  study  you  do,  consider  at  least  the  possibility  of 
sticking  with  modern  equipment,  but  look  at  equipment  that  results  in  complex 
tasks  and  other  equipment.  Although  it  may  be  complex,  the  tasks,  in  fact, 
are  not  that  difficult.  Both  of  those  areas  appear  to  me  to  be  primarily  in 
fhe  electronics  area.  I think  you  should  also  look  into  a mechanical  system. 

I would  look  at  three  test  cases.  I would  look  at  a complex  task  electronic 
system,  a not-so-complex  task  electronic  system,  and  a mechanical  system. 

If  you're  rich,  also  look  at  a system  that  involves  an  integration  of  mechan- 
ical and  electrical  systems  that  uses  "high-powered"  electronics.  I would 
suggest  that  you  analyze  and  test  proceduralized  troubleshooting  vs.  deductive 
troubleshooting  for  those  kinds  of  systems.  At  least  analyze  it  to  see 
whether  we  have  evidence  that  we  can  produce  troubleshooting  formats  for 
them.  Regardless  of  the  complexity  of  the  equipment,  if  the  equipment  tasks 
are  manageable  and  task  breakouts  are  limited,  fully  proceduralized  aids 
can  work.  As  the  number  of  things  that  can  go  wrong  goes  up,  fully  proced- 
uralized troubleshooting  aids  are  not  a viable  solution. 

Mr.  Johnson:  I'd  like  to  comment  on  Reid's  discussion  on  the  acceptance 

of  JPAs,  not  so  much  at  the  user  level  but  at  the  program  manager's  level.  I 
think  that  that  is  the  area  we  are  going  to  have  to  concentrate  on  if  we  are 
going  to  realize  the  potential  of  JPAs.  The  tool  that  appears  to  be  most 
available  is  the  concept  that  Ted  is  working  on  which  could  be  built  into 
a program  manager's  guide.  This  program  manager's  guide  should  identify 
what  is  required  for  a system  in  terms  of  time  (year  one,  year  two,  etc.) 
and  identify  some  costs  in  each  year.  As  you  progress  through  your  program  and 
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get  more  experience  on  what  kind  of  data  you  need  and  the  kind  of  maintenance 
manuals,  the  PM  can  go  ahead  and  selectively  commit  himself  to  buy  one  thing 
and  then  in  year  three  you  can  take  a better  look  and  buy  another.  In  this 
way  we've  accomplished  several  things.  First,  you  have  given  him  a tool  that 
he  can  actually  use  to  cost  out  his  program  and  you  can  show  him  that  you  can 
reduce  total  program  costs.  You  don't  want  to  commit  all  your  dollars  at 
one  time.  If  you  do,  towards  the  end  of  the  program,  you  will  find  that  a lot 
of  dollars  you  committed  earlier  were  just  wasted  because  the  material  was 
bought  too  soon.  The  program  manager  is  taking  a very  big  chance  by  buying 
too  early  in  the  game. 

Mr.  Joyce:  Ted  presently  has  a lot  of  information  to  put  into  the  program 

manager '8  guide;  but,  to  use  Jack's  refinement  of  the  idea,  I think  we  as  the 
research  community  really  do  have  to  start  postulating  some  bigger  pictures  of 
the  system.  Run  numbers  and  concepts  through  to  establish  that  they  are 
economically  feasible  so  we  will  have  something  to  sell  to  the  program  managers. 
I don't  really  care  whether  that  is  driving  the  system  or  just  providing  enough 
input  so  the  system  would  be  willing  to  change  for  itself.  I think  we  do  have 
to  have  the  who'le  personnel  system  view  before  we  are  going  to  be  in  a posi- 
tion to  have  something  that  is  really  saleable.  Our  arguments  over  the  years 
have  been  the  use  of  fully  proceduralized  kinds  of  things.  The  use  of  fully 
proceduralized  aids  has  the  potential  for  substantial  saving  in  training.  Our 
experiments  have  been  snapshots  that  demonstrated  that  we  can  produce  aids, 
some  of  them  that  work,  some  of  them  that  don't  work  so  well.  People  usually 
can  perform  with  the  aids  pretty  well.  I don't  feel  confident  that  we  have 
something  saleable,  but  I haven't  spent  very  much  time  trying  to  conceptualize 
it  for  myself. 

Mr.  Post:  I guess  I'd  like  to  make  two  points.  One  about  the  production 

of  technical  data,  whether  its  fully  proceduralized  or  whether  it's  the  deduc- 
tive type  or  some  other  type.  I guess  I feel  that  one  of  the  biggest  problems 
in  the  production  process  has  to  do  with  the  accuracy  and  timeliness  of 
delivery  and  I wish  I had  a solution  to  it.  I think  we  need  an  entirely  new 
notion  of  validation  and  verification.  I think  we  might  look  to  the  NTIPP 
program  and  the  work  that  Hughes  is  doing  and  hope  that  they  come  up  with 
some  good  notions  as  to  how  to  improve  the  process.  I think  maybe  we  ought 
to  give  some  thought  to  a complete  change.  In  the  automotive  industry,  for 
example,  if  the  factories  seem  unable  to  get  cars  out  in  good  shape,  they 
transfer  some  of  the  responsibility  to  the  dealerships.  Maybe  something  like 
that  might  take  place  in  the  production  and  use  process.  We  might  transfer 
some  of  the  responsibility  for  verifying  aids  to  the  technicians  who  use  them. 
After  all,  that  is  happening  now.  They  are  the  people  that  identify  all  the 
errors  and  omissions.  But  we  ought  to  give  them  the  wherewithall  to  do  that. 

We  ought  to  slick  up  the  feedback  process  a great  deal  more  than  it  is  right 
now.  That  gives  me  an  entree  to  the  second  thought  which  is  acceptability  by 
the  user  and  if  in  fact  we  can  get  him  to  be  a cooperating  and  productive  part 
of  the  accuracy  improvement  program.  It  is  going  to  be  his  aid  a great  deal 
more  than  our  aid.  Also,  we  have  to  stop  looking  at  the  using  community  as 
solely  inexperienced  or  experienced.  We,  in  fact,  have  gradations  of  experi- 
ence and  we  ought  to  have  aids  that  meet  the  needs  of  all  of  these  people  out 
in  the  field.  I think  they  have  to  be  progressive  and  the  aid  at  the  lowest 
level  not  only  has  to  get  productive  labor  but  it  has  to  transition  the  gent 
to  the  next  plateau.  The  notion  that  Hal  Booher  had  about  a family  of  aids 
is  a vety  viable  and  needed  concept. 


COL  Grossel:  I want  to  believe  and  I keep  hearing  from  other  places  in 

addition  to  here  that  the  so-called  JPA  technology  base  is  here  and  ready  to 
be  implemented.  There  are  still  a lot  of  things  that  have  to  be  worked  out. 

For  example,  you're  talking  about  implementation.  What  are  you  going  to  imple- 
ment? In  order  to  implement  something  you  have  to  have  a spec.  You  need  a 
data  item  description.  I hear  a lot  of  talk  about  the  need  for  task  analysis 
and  other  products.  Some  products  called  for  in  the  past  may  or  may  not  be 
necessary.  I don't  think  there  is  any  agreement  as  to  what  products,  what 
single  specs,  what  standards,  what  data  item  should  be  put  on  contract.  It's 
nice  to  have  a lot  of  people  promoting  different  concepts  which  may  have  the 
same  underlying  requirements  but  until  we  can  get  agreement  on  some  small 
number  of  specs  and  other  standards  to  support  underlying  processes  I don't 
see  the  system  as  really  being  there  for  implementation.  I'm  not  saying  we 
don't  need  the  guides  for  program  managers;  we  do  need  them!  It’s  even  more 
basic  than  that.  Ed's  work  for  the  Army  looked  across  the  commodities,  and 
essentially  all  these  commodities  could  use  essentially  the  same  spec. 

At  my  level  I'm  looking  for  some  attempt  to  get  more  tech  manual  and  spec 
standards  established.  This  is  an  important  fact  for  evaluations  also  because 
you  can  go  ahead  and  evaluate  technology.  When  you  do  that,  you  are  evaluating 
a spec,  and  the  necessary  things  that  go  with  that  spec.  But  the  Army  may  not 
buy  it  or  the  Air  Force.  I think  to  make  real  progress  in  this  area  we  have 
got  to  get  the  services  together  and  get  more  agreement.  It's  not  industry's 
fault.  I think  it's  the  Department  of  Defense's  fault.  I think  it's  even 
the  services'  fault  because  they  have  their  own  interests  and  they  have  their 
own  operating  environments  and  so  on.  DoD  hasn't  clamped  down  hard  enough 
on  standardization. 

Dr.  Shriver:  We,  the  human  resources  research  community,  are  in  an  imple- 

mentation area  we  are  not  prepared  to  deal  with.  We  have  a technological  base 
but  not  enough  people  who  have  the  technology  to  get  implementation  accom- 
plished correctly.  As  Jack  says,  that  is  all  part  of  the  base,  whether  we've 
got  It  or  not  is  the  question.  The  result,  though,  is  many  errors  in  imple- 
mentation or  products;  and,  as  Reid  says,  we  need  to  get  a bigger  implementa- 
tion picture  for  JPA  products  or  we'll  wind  up  at  road  blocks  like  we  can't 
pass  promotion  tests,  no  growth  on  the  job,  etc.  We  don't  have  the  technical 
manpower  to  handle  what's  happening.  We  need  to  get  bigger  and  I think  this 
is  a now  rock  and  hard  place  for  us  which  even  research  funding  doesn't  ade- 
quately recognize.  It  is  research  on  Implementation  itself  in  its  funny  new 
kind  of  place.  The  bridge  from  research  to  implementation  isn't  there. 

Research  funds  are  now  being  cut  off  on  the  grounds  that  we  already  know  that 
JPAs  work,  and  we  don't  need  any  more  research  to  demonstrate  that.  It's 
true  that  we  don't  need  more  research  in  the  old  structured  mold.  We  desperatel 
need  research  on  a larger  context  that  has  been  opened  up  by  partial  implemen- 
tation. I think  this  is  an  immediate,  desperate  need.  We  have  not  built  up 
the  logic  and  sales  compaign  to  explain  the  new  rock  and  hard  place  problems 
to  the  controllers  and  research  funders.  Our  old  pitches  are  for  diffeient 
purposes  and  Joyce  was  talking  about  them.  They  have  merely  led  to  the  new 
problems.  We  are  Just  gathering  the  pieces  of  a new  pitch  and  we're  begin- 
ning to  put  it  together  here.  Funds  may  dry  up  before  we  can  get  the  pitch 
and  an  audience  together.  I think  Roger  Grossel's  presence  here  is  very 
important.  He's  at  the  money  priority  control  spickets  and  he's  hearing  the 
beginnings  of  a new  strategy  being  put  together  in  this  meeting.  I sincerely 
hope  that  we  can  refocus  the  research  community  on  the  next  generation  of 
probi ems. 


Dr.  Collins:  One  area  that  interests  me  is  the  test  and  evaluation  area 

which  relates  to  demonstration  and  implementation.  I am  only  aware  of  one 
computerized  approach  that  has  been  developed  by  NAVAIR  which  was  reported  at 
the  sixth  IEA  meeting  last  year  for  the  assessment  of  JPAs,  and  I think  that 
is  part  of  the  technology  base.  That  was  the  work  that  was  done  at  North 
Carolina  State  for  NAVAIR.  There  may  be  other  capabilities  but  that  is  the 
only  one  of  which  I am  aware.  We  all  know  that  there  is  supposed  to  be  cut-off 
scores  for  people  to  go  to  schools,  to  get  into  occupations,  and  you  can  design 
JPAs  around  that  cut-off  score.  In  the  real  world,  if  you  look  at  the  distri- 
bution of  mental  levels,  you'll  find  a rather  wide  dispersion,  and  whether  or 

not  the  JPAs  are  adequate  to  handle  that  dispersion  as  compared  to  the  label 
he  may  have  of  an  MOS  or  an  NEC  is  a very  legitimate  question  with  regard  to 

utilization  and  implementation.  I'd  like  to  make  one  other  modest  suggestion. 

I just  finished  a report  for  NAVPERSRANDCEN  in  the  area  of  technology  applica- 
tions. It  included  a review  of  the  literature  and  development  of  a set  of 
guidelines  for  technology  application  where  I look  at  24  specific  end  products 
of  the  center  and  the  fleet  support  area.  I looked  at  the  defining  character- 
istics of  those  end  products  and  what  were  the  communication  networks  associated 
with  planning  development  and  implementation.  Follow-on  work  to  that  will 
be  a definitive  network  analysis  in  the  training  area.  Perhaps  there  is  a 
requirement  here  to  have  some  kind  of  mapping  of  the  process  or  planning 
development  and  implementation  that  relates  to  this  kind  of  set  of  inter- 
faces, organizational  and  other  dimensions. 

Mr.  Post:  I think  that's  a strong  point,  in  my  view  anyhow.  The  idea  to 

look  at  a larger  portion  of  the  R&D  implementation  sequence  started  with  Merle 
Malehorn.  I think  some  of  the  points  he  made  in  his  paper  of  2 years  ago  are 
still  very  valid.  We  have  to  go  ahead  and  look  at  those  kinds  of  things. 

Mr.  Joyce:  We  need  to  come  up  with  a better  validation/verification  pro- 

cess and  also  answer  the  problems  of  delivering  to  the  user.  The  project  that 
I did  involved  15,000  pages  of  fully  proceduralized  troubleshooting.  We  got 
the  average  amount  validated  (10-15%)  before  all  of  the  systems,  including 
some  that  were  still  being  assembled  in  the  factory,  were  delivered  to  the 
user.  At  last  report  the  North  Vietnamese  were  still  trying  to  verify  them. 

SESSION  4 — Thursday  afternoon,  February  24,  1977 

The  selected  discussion  comments  below  followed  the  papers  of  Dr.  Shriver, 
"New  Direction  for  Information  Transfer  Research  in  Maintenance  Jobs,"  and 
Mr.  Klesch,  "Implementation  of  the  JPA/ Job-oriented  Training  Approach  to 
Maintenance:  The  Impact  on  Personnel  Systems." 

Dr.  Shriver:  You  have  to  go  ahead  and  make  the  personnel  reduction. 

The  studies  show  that  increased  performance  is  there.  That's  got  to  be  trans- 
lated into  reduced  personnel.  The  other  payoff  is  reduction  of  training  time. 
That's  nonproductive  time.  You  can  use  OJT  where  he  is  working  on  the  job 
while  turning  out  some  products  on  the  job. 
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Dr.  Collins:  What  about  preserving  the  life  of  the  equipment?  I'm  saying 

that  a man  who  does  an  effective  maintenance  job  perhaps  could  save  an  airplane. 

Mr.  Post:  Better  operation  or  better  maintenance  of  a smaller  buy  of 

expensive  hardware  is  a possibility.  We  are  dealing  with  the  front  end  now. 

COL  Grossel:  That  is  something  the  operational  commanders  will  never 

buy.  They  will  buy,  "reduce  my  personnel  if  you  can  give  me  more  airplanes." 

Mr.  Post:  I get  nervous,  Ed,  when  you  give  the  big  global  figures  that 

we  have  a quarter  of  a million  in  this  or  that  area  (for  savings  in  personnel 
reduction).  I have  looked  down  a list  of  data  items  and  one  of  the  things 
listed  there  was  the  amount  of  boiler  technician's  time  spent  on  scheduled 
maintenance.  It  was  about  2 percent  of  his  time,  but  we  are  not  going  to 
eliminate  that  man. 

Dr.  Booher:  I would  like  to  go  back  to  some  of  the  questions  we  have  been 
sliding  over.  I get  the  impression  that  there  are  only  two  alternatives  being 
offered  so  far  in  this  training  area.  Either  we  cut  it  (training)  out  entirely 
and  have  it  (training)  supported  by  a book  with  the  supervisor  there  to  aid  and 
assist,  but  he  won't  be  an  instructor;  or,  we  have  the  conventional  type  train- 
ing system. 

Dr.  Collins:  I was  interested  in  what  kinds  of  troubleshooting  strategies 

were  preferred  and  being  designed  into  the  fully  proceduralized  aids. 

Mr.  Johnson:  I go  back  to  the  various  specifications  that  tell  you  how  to 

get  fully  proceduralized  troubleshooting.  It's  a big  thick  book.  There's  one 
paragraph  in  there  that  says  find  youself  an  experienced  technician  and  let 
him  do  the  actual  troubleshooting. 

Dr.  Shriver:  He  is  taught  an  overview  of  the  thing,  at  the  block  diagram 

level.  Why  he  would  stop  going  down  the  (decision)  chain  and  switch  to  another 
(branch)  will  probably  be  a function  of  the  next  check  that  he  has  to  make. 

If  this  one  is  going  to  take  him  half  an  hour  to  set  up,  he  might  as  well  skip 
that  one,  go  over  here  and  make  a 2-minute  check,  remembering  that  it  still 
may  be  down  there  (the  first  branch) . 

Mr.  Johnson:  There  is  no  standard  means  of  developing  troubleshooting. 

One  of  the  better  techniques  that  is  very  costly  is  maintenance  dependency 
charts.  With  a good  maintenance  dependency  chart,  you  know  what  your  inputs 
are,  you  know  what  your  dependencies  are,  you  know  what  your  test  specifica- 
tions are,  and  you  know  what  your  output  should  be. 

Dr.  Collins:  I was  interested  in  this  concept  of  the  naive  man  who  is 

given  the  information  to  go  and  do  diagnostic  troubleshooting.  How  general- 
izable  are  the  skills  he  was  learning  or  using  with  this  JPA  to  other  classes 
of  similar  problems?  How  equipment  specific  do  they  have  to  be? 
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Dr.  Shriver:  He  learns  the  test  equipment,  how  to  open  the  latches,  how 

to  cheat  the  Interlocks,  and  another  system  may  have  some  similarities.  He 
learns  all  the  junk  stuff,  which  is  very  useful  to  him  since  it  frees  his  head 
for  other  things. 

Dr.  Blanchard:  Jack,  are  you  getting  at  the  Bayesian  strategies,  the  cog- 

nitive processes  in  troubleshooting  that  people  like  Joe  Rigney  tried  to  model? 

Dr.  Collins:  That's  part  of  it.  I was  trying  to  get  a better  under- 

standing of  what  was  being  designed  into  these  capabilities  for  several  reasons, 
one  being  user  acceptance,  also  generalizability.  There  are  a variety  of  other 
kinds  of  criteria  that  one  might  want  to  think  about.  My  impression  is  that 
we  are  talking  about  signal  tracking.  Signal  tracking  is  the  primary  strategy 
being  designed  here.  There  was  also  a suggestion  that  once  you  do  that, 
through  some  kind  of  engineering  analysis,  the  people  developing  these 
(analyses)  hopefully  have  some  optimum  strategy  that  is  more  efficient  than 
other  strategies. 

Mr.  Klesch:  When  you  don't  know  what  is  wrong,  the  situation  looks  dif- 

ferent. What  used  to  be  a puppy  dog,  when  I fail  it  or  when  I take  a circuit 
board  or  card  out,  becomes  a tiger  in  the  real  world.  When  they  bring  one  in 
off  the  line,  we  all  stand  around,  and  now  we  are  going  to  put  our  magic  book 
to  work.  I'm  not  sure  if  my  legs  are  going  to  be  lowered  or  what.  We  took  that 
same  procedure  over  to  Lockheed  and  their  equipment  was  running  a little  bit 
differently.  The  power  supply  varied  a little  bit  and  our  tolerances  for  the 

windows  were  much  too  narrow.  It  was  only  because  a qualified  technician  was 

there  that  we  found  the  fault.  A dummy  using  this  JPA  would  go  off  the  track. 
But  the  technician  knew  that  the  window  should  have  been  wider.  He  proceeded 
down  and  found  the  fault.  He  demonstrated  to  us  that  there  are  a lot  of  avail- 
able cues  (not  in  the  book)  used  by  the  conventionally  trained  man. 

COL  Grossel:  Problems  can  be  overcome  by  providing  better  JPAs.  When 

are  our  JPAs  good  enough  to  implement  and  when  do  we  go  out  on  the  limb  to  be- 
come operational?  Are  you  in  the  Air  Force  ready  to  put  your  case  on  the  line 

and  say  there  is  our  specification,  here  are  the  standards  and  DIDs  and  CDRLs? 
Put  that  in  the  contract  and  that  will  give  you  a good  JPA? 

Mr.  Klesch:  Say  we  have  a new  HF  radio.  To  JPA  or  not  to  JPA,  that  is 

the  question.  I think  we  can  do  that.  But  if  he  comes  to  me  with  an  F106 
firecontrol  system,  I don't  think  I can  do  JPAs  on  that  system.  I don't 
think  anybody  can. 

Dr.  Collins:  Are  you  saying  it's  the  technology  of  JPAs  that's  making 

you  reluctant  or  the  absence  of  a good  technical  base  about  the  system  or  the 
hardware? 

Dr.  Shriver:  The  JPA  for  troubleshooting  is  inherently  inefficient. 

Mr.  Post:  Troubleshooting  JPAs  are  based  on  the  checkout  procedure  and 

it  may  not  be  right. 

Mr.  Klesch:  On  a missile  cooling  unit,  no  matter  even  if  you  took  the 

least  efficient  approach,  there  are  only  X number  of  things  to  check.  Ineffi- 
ciency doesn't  matter. 
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Dr.  Collins:  Are  there  any  guidelines  that  you  can  see,  John,  that 

would  help  to  define  the  upper  limits  if  there  are  any  upper  limits?  Are 
there  classes  of  equipments  that  come  to  mind? 

Mr.  Klesch:  You  almost  need  to  do  a hierarchical  equipment  inventory  of 

what  you've  got.  I don't  know  how  to  answer  the  problem  of  do  we  only  go  so 
far  or  what  happens  when  we  get  into  a strange  problem.  If  you  have  it  in 
that  same  specialty,  man  X comes  in  and  is  only  allowed  to  troubleshoot  down 
so  far.  If  you  think  man  Y is  going  to  be  the  higher  cat  within  that  same 
specialty,  I think  you're  asking  for  trouble.  There  almost  has  to  be  two 
guys.  Can  we  train  a medium  aptitude  guy  to  train  second  enlistments  a 
higher  electronics?  The  data  indicate  in  the  past  that  you  can't. 

Mr.  Johnson:  Every  system  is  a combination  of  various  components  and  not 

a mass  of  electronics  just  shoved  in.  We  don't  manufacture  things  that  way 
anymore.  Nobody  does. 

Dr.  Shriver:  Every  system  can  be  analyzed  into  parts. 

Mr.  Johnson:  Yes,  otherwise  you  couldn't  design  it.  You  couldn't  even 

check  it  in  the  shop.  At  Westinghouse  10  years  ago  they  brought  in  a new 
numerical  control  wiring  machine.  It  was  a marvel  to  behold.  They  just  threw 
wires  in  there  all  over  the  place  and  if  it  worked,  it  worked.  But  if  it  didn't 
work,  you  could  forget  about  it.  Nobody  could  fix  it.  They  had  to  do  away  with 
it  finally  because  of  maintainability  requirements. 

Dr.  Collin^:  Are  you  saying,  Frank,  that  there  are  no  systems  for  which 

a fully  proceduralized  JPA  could  be  developed  for  electronics? 

Mr.  Johnson:  If  you  are  time  bound,  you  may  not  have  time  to  go  through 

many  of  these  steps. 

Mr.  Klesch:  I would  like  to  see  an  automatic  test  equipment  vs.  manual 

approach  examined  thoroughly.  Where  does  that  tradeoff  occur? 

Mr.  Post:  I think  that  is  an  excellent  point  and  it's  something  that  I 

describe  as,  if  you  have  a buck  to  get  the  kind  of  performance  needed  out  of 
the  technician  you  have  to  work  with,  how  do  you  spend  that  buck?  Do  you  spend 
it  60c  on  training  and  20c  on  JPA?  Or  is  there  a different  balance?  I don't 
know  a scintilla  of  evidence  to  support  any  such  analyses. 

Mr.  Post:  I think  this  business  of  considering  the  tech  manual  or  the 

JPA  or  MDC  or  whatever  it  is  as  being  developed  only  when  it  comes  time  to 
deliver  the  equipment  is  just  untenable.  I think  it's  a dynamic  process  that 
goes  throughout  development  of  the  system.  It  has  to  be  updated,  corrected, 
refined,  and  modified  with  heavy  inputs  from  the  user.  I get  the  idea  that 
the  articulation  between  the  training  process,  the  process  by  which  training 
materials  are  developed,  and  the  process  by  which  JPAs  or  tech  manuals  are 
developed  does  not  exist.  There  is  no  articulation  between  them.  I don't 
think  the  tech  manual  people  talk  the  training  language  and  I think  the 
reverse  is  true  also.  When  we  modify  the  training  program  and  complement  it 
with  JPAs  in  the  field,  it  seems  like  we  are  coming  from  two  different  arenas. 
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Dr.  Collins:  One  of  the  most  basic  principles  in  all  training  is  that 

you  design  training  as  much  as  possible  around  the  previous  learning  and 
learning  style  of  the  individual  to  fit  that  kind  of  interface. 

SESSION  5 — Friday  morning,  February  25,  1977 

The  following  comments  are  selected  concluding  remarks. 

Dr.  Blanchard:  We  need  to  incorporate  the  JPA  with  some  form  of  training 
support  so  that  the  individual  can’t  just  be  confronted  with  the  JPA  at  some 
point  and  left  to  fend  for  himself.  What  ideas  do  we  have,  what  experiences 
have  we  had  concerning  the  need  and  the  manner  in  which  training  packages 
have  been  incorporated  with  the  introduction  of  JPAs? 

Mr.  Klesch:  There  are  three  ways  that  you  can  address  this  training 

situation.  The  first  one  is  that,  given  some  X amount  of  training  you  now 
create  JPA,  or  (second)  given  you  have  a JPA,  you  now  make  some  training. 

It  now  seems  that  (the  third  way)  doing  them  together,  makes  the  most  sense. 

Mr.  Post:  Concerning  the  interaction  of  JPA  and  the  training  the  ulti- 
mate user  will  receive,  I'd  like  to  broaden  that  concept  a little  bit  and 
talk  about  the  process  by  which  JPAs  are  selected  and  developed.  JPA  develop- 
ment should  interact  more  fully  with  the  rest  of  the  personnel  and  training  sub 
system.  Job  design  is  a key  aspect  of  the  entire  personnel  subsystem.  I'm 
asking  for  a full  disclosure,  interactive  kind  of  process.  We  ought  to  bring 
in  the  personnel  turnover  that  we're  expecting.  From  the  personnel  turnover 
aspect,  we  can  say  we  are  anticipating  certain  things.  In  response  to  this, 
we  in  the  JPA  business  are  going  to  counter  with  this  approach.  It's  not 
just  training  and  JPAs.  It's  more  than  that.  With  respect  to  the  mainte- 
nance workload,  arriving  at  the  same  time  is  a consideration.  I think  that 
has  major  implications  for  organizational  structure  and  job  design  as  well 
as  JPAs. 


Dr.  Collins:  It  would  be  useful  to  distinguish  the  general  problems 

associated  with  the  introduction  of  JPAs  from  the  problems  of  running  a demon- 
stration project.  One  of  the  dangers  in  any  new  program  of  this  kind  is  to 
be  too  ambitious.  If  you're  going  to  conduct  a series  of  demonstrations  or 
experiments  under  this  program,  it  seems  you  have  essentially  three  objectives. 
One  is  to  demonstrate  the  utility  or  the  value  of  some  technology  that  is  avail 
able.  The  second  one  is  to  solve  the  specific  operational  problems  in  the 
process.  Thirdly,  is  to  impact  on  the  system  in  some  general  way  whether 
it  is  the  personnel  system  or  the  training  system  or  both.  One  of  the  things 
that  needs  to  be  considered  is  how  much  of  this  general  goal  of  interaction 
can  you  really  try  to  encompass  within  the  NAVPERSRANDCEN  project.  I agree 
that  it  is  important  to  think  about  the  impact  on  the  personnel  system.  You 
may  find  that  tackling  the  interaction  process  between  JPAs  and  training  is  all 
you  can  hope  to  do. 

Mr.  Johnson:  I'd  like  to  suggest  that  the  area  of  engineering  drawings 

plays  an  important  part  in  bringing  together  the  package.  If  JPA  technology 
is  going  to  offer  anything,  it  has  to  tie  in  training,  tie  in  basic  Job- 
oriented  tasks,  and  give  the  maintainer  the  tools  he  needs  to  perform  his 
actual  maintenance.  A way  of  implementing  a new  program  is  to  tie  it  in  to 
the  release  of  engineering  drawings,  to  the  engineering  production  process. 
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That  is  the  very  beginning  of  all  training,  all  maintenance,  and  we  never 
really  address  that  particular  facet  of  it  in  my  experience.  We  should 
investigate  the  possibility  (during  the  early  acquisition/design  process  by 
the  contractor)  of  identifying  in  the  work  breakdown  structure  exactly  what  is 
required  in  the  way  of  maintenance  training  for  each  assembly  as  it  is  released 
for  engineering  production  drawing  to  the  shop  for  manufacturing.  If  you  do 
this,  you  gain  a tremendous  number  of  advantages.  The  biggest  advantage  is 
manageable  chunks  of  work.  Another  powerful  tool  is  I can  schedule  it  in  the 
same  way  that  engineering  design  groups  schedule  right  out  of  the  drafting 
room  into  the  shop  and  released  for  manufacture.  I can  schedule  my  training 
requirements,  maintenance  analysis,  and  JPA  production.  This  is  a specific 
way  of  doing  the  things  that  we're  talking  about. 

Dr.  Collins:  People  will  argue  if  you  want  to  go  from  a theory-based 

training  program  to  one  that's  highly  job  structured.  Do  that  analysis  on 
the  old  system  first  and  add  your  changes.  Systems  don't  change  that  much. 

I think  people  tend  to  go  to  an  analysis  of  the  old  hardware  and  then  intro- 
duce the  new  changes  into  that,  and  go  to  drawings  and  do  an  analysis. 

Dr.  Shriver:  JPA  is  a code  word  to  refer  to  a procedure.  JPA  is  also 

generic  but  you  don't  use  it  generically  because  when  you  use  it  generically 
there  are  lots  of  new  concepts  that  have  conceptual  material  in  them.  FORE- 
CAST is  the  one  that  has  the  most  of  it,  i.e.,  concepts  put  on  paper.  JPA 
has  been  used  as  a code  word  to  refer  to  a click,  click,  click  procedure, 
and  not  in  the  generic  sense.  One  way  of  dealing  with  concepts  is  to  lay  it 
out  in  block  diagram  form  so  you  can  see  everything.  Work  breakdown  structure 
can  be  thought  oi  as  slicing  the  whole  thing  down  into  pieces.  It  is  the  same 
process  for  a block  diagram,  slice  it  up  into  pieces.  Conceptually  people  can 
understand  how  all  those  pieces  are  put  together.  There  is  a dependency 
between  them.  They  can  learn  that.  The  old  timers,  who  understand  a system, 
break  it  up  in  their  heads  so  they  understand  it  functionally  and  then  they 
understand  the  schematics. 

Dr.  Booher:  Could  we  give  them  a much  shortened  course,  say  ten  percent 

of  what  is  normally  given?  Give  them  some  basic  skills  training,  and  along 
with  this,  some  ability  to  read  functional  flow  blocks,  and  be  able  to  utilize 
written  material  in  much  shorter  training  session? 

Dr.  Shriver:  Yes,  if  you  are  talking  troubleshooting,  which  is  only 

half  the  world. 

COL  Grossel:  One  of  the  words  Hall  used  was  common  use  test  equipment. 

One  of  the  things  that  this  1TDT  specification  breaks  out  is  common  task  in  a 
separate  job  performance  guide.  Common  tasks  could  include  common  use  test 
equipment . 

Mr.  Post:  I think  it  would  be  useful  to  have  a glossary  to  try  to  at 

least  identify  some  of  these  things. 

Dr.  Blanchard:  Our  goals  are:  (1)  we  have  to  demonstrate  a technology; 

(2)  we  have  to  try  to  make  some  hay  by  solving  an  operational  problem;  and 

(3)  at  the  same  time  we  have  to  assess  JPA  impact  on  the  larger  system,  i.e., 
the  personnel  and  training  you  are  trying  to  integrate  into  the  system,  and 
the  development  process.  With  those  three  general  things  in  mind,  considering 
our  resources,  we  have  an  optimization  problem. 
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